2023-07-20T01:01:15.096 INFO:root:teuthology version: 0.0.1.dev153+g79491cb 2023-07-20T01:01:15.098 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344241 --verbose --owner scheduled_pdonnell@teuthology --name pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi --description fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} -- /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344241/orig.config.yaml 2023-07-20T01:01:15.168 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-20T01:01:15.280 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344241 branch: wip-pdonnell-testing-20230719.201220 description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} email: pdonnell@redhat.com first_in_suite: false job_id: '7344241' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-pdonnell-testing-20230719.201220 ceph: cephfs: max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd heartbeat grace: 60 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - SLOW_OPS - slow request sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 kclient: client.0: mount_subvol_num: 1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20230719.201220 sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 6483 sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 sleep_before_teardown: 0 subset: 383/512 suite: fs:workload suite_branch: wip-pdonnell-testing-20230719.201220 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi060.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi136.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi183.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - ceph-fuse: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1M - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6M3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 79491cb69dd37a1eafb93b3471998174bdfa2fd9 timestamp: 2023-07-20_00:39:06 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-07-20T01:01:15.280 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa; will attempt to use it 2023-07-20T01:01:15.281 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks 2023-07-20T01:01:15.281 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-07-20T01:01:15.284 INFO:teuthology.task.internal:Checking packages... 2023-07-20T01:01:15.316 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '1b0b2eb005a7c51d97a788b9900d511e1394b537' 2023-07-20T01:01:15.316 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-07-20T01:01:15.316 INFO:teuthology.packaging:ref: None 2023-07-20T01:01:15.317 INFO:teuthology.packaging:tag: None 2023-07-20T01:01:15.317 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20230719.201220 2023-07-20T01:01:15.317 INFO:teuthology.packaging:sha1: 1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T01:01:15.317 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-pdonnell-testing-20230719.201220 2023-07-20T01:01:15.826 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-5040.g1b0b2eb0 2023-07-20T01:01:15.827 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-07-20T01:01:15.833 INFO:teuthology.task.internal:no buildpackages task found 2023-07-20T01:01:15.833 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-07-20T01:01:15.840 INFO:teuthology.task.internal:Saving configuration 2023-07-20T01:01:15.865 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-07-20T01:01:15.872 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-20T01:01:15.900 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi060.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344241', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 00:56:00.541788', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-20T01:01:15.927 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344241', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 00:56:00.545106', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-20T01:01:15.955 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344241', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 00:56:00.543826', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-20T01:01:15.955 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-07-20T01:01:15.962 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2023-07-20T01:01:15.962 INFO:teuthology.task.internal:roles: ubuntu@smithi136.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2023-07-20T01:01:15.962 INFO:teuthology.task.internal:roles: ubuntu@smithi183.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2023-07-20T01:01:15.963 INFO:teuthology.run_tasks:Running task console_log... 2023-07-20T01:01:16.084 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbae2dab430>, signals=[15]) 2023-07-20T01:01:16.086 INFO:teuthology.run_tasks:Running task internal.connect... 2023-07-20T01:01:16.094 INFO:teuthology.task.internal:Opening connections... 2023-07-20T01:01:16.094 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi060.front.sepia.ceph.com 2023-07-20T01:01:16.097 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T01:01:16.169 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi136.front.sepia.ceph.com 2023-07-20T01:01:16.170 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T01:01:16.241 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi183.front.sepia.ceph.com 2023-07-20T01:01:16.243 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T01:01:16.309 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-07-20T01:01:16.316 DEBUG:teuthology.orchestra.run.smithi060:> uname -m 2023-07-20T01:01:16.337 INFO:teuthology.orchestra.run.smithi060.stdout:x86_64 2023-07-20T01:01:16.338 DEBUG:teuthology.orchestra.run.smithi060:> cat /etc/os-release 2023-07-20T01:01:16.396 INFO:teuthology.orchestra.run.smithi060.stdout:NAME="Red Hat Enterprise Linux" 2023-07-20T01:01:16.397 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION="8.6 (Ootpa)" 2023-07-20T01:01:16.397 INFO:teuthology.orchestra.run.smithi060.stdout:ID="rhel" 2023-07-20T01:01:16.397 INFO:teuthology.orchestra.run.smithi060.stdout:ID_LIKE="fedora" 2023-07-20T01:01:16.397 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_ID="8.6" 2023-07-20T01:01:16.397 INFO:teuthology.orchestra.run.smithi060.stdout:PLATFORM_ID="platform:el8" 2023-07-20T01:01:16.398 INFO:teuthology.orchestra.run.smithi060.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-20T01:01:16.398 INFO:teuthology.orchestra.run.smithi060.stdout:ANSI_COLOR="0;31" 2023-07-20T01:01:16.398 INFO:teuthology.orchestra.run.smithi060.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-20T01:01:16.398 INFO:teuthology.orchestra.run.smithi060.stdout:HOME_URL="https://www.redhat.com/" 2023-07-20T01:01:16.398 INFO:teuthology.orchestra.run.smithi060.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-20T01:01:16.398 INFO:teuthology.orchestra.run.smithi060.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-20T01:01:16.399 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:01:16.399 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-20T01:01:16.399 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-20T01:01:16.399 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-20T01:01:16.399 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-20T01:01:16.400 INFO:teuthology.lock.ops:Updating smithi060.front.sepia.ceph.com on lock server 2023-07-20T01:01:16.424 DEBUG:teuthology.orchestra.run.smithi136:> uname -m 2023-07-20T01:01:16.444 INFO:teuthology.orchestra.run.smithi136.stdout:x86_64 2023-07-20T01:01:16.444 DEBUG:teuthology.orchestra.run.smithi136:> cat /etc/os-release 2023-07-20T01:01:16.507 INFO:teuthology.orchestra.run.smithi136.stdout:NAME="Red Hat Enterprise Linux" 2023-07-20T01:01:16.508 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION="8.6 (Ootpa)" 2023-07-20T01:01:16.508 INFO:teuthology.orchestra.run.smithi136.stdout:ID="rhel" 2023-07-20T01:01:16.508 INFO:teuthology.orchestra.run.smithi136.stdout:ID_LIKE="fedora" 2023-07-20T01:01:16.508 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_ID="8.6" 2023-07-20T01:01:16.508 INFO:teuthology.orchestra.run.smithi136.stdout:PLATFORM_ID="platform:el8" 2023-07-20T01:01:16.508 INFO:teuthology.orchestra.run.smithi136.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-20T01:01:16.509 INFO:teuthology.orchestra.run.smithi136.stdout:ANSI_COLOR="0;31" 2023-07-20T01:01:16.509 INFO:teuthology.orchestra.run.smithi136.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-20T01:01:16.509 INFO:teuthology.orchestra.run.smithi136.stdout:HOME_URL="https://www.redhat.com/" 2023-07-20T01:01:16.509 INFO:teuthology.orchestra.run.smithi136.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-20T01:01:16.509 INFO:teuthology.orchestra.run.smithi136.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-20T01:01:16.509 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-07-20T01:01:16.509 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-20T01:01:16.509 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-20T01:01:16.510 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-20T01:01:16.510 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-20T01:01:16.510 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server 2023-07-20T01:01:16.532 DEBUG:teuthology.orchestra.run.smithi183:> uname -m 2023-07-20T01:01:16.553 INFO:teuthology.orchestra.run.smithi183.stdout:x86_64 2023-07-20T01:01:16.553 DEBUG:teuthology.orchestra.run.smithi183:> cat /etc/os-release 2023-07-20T01:01:16.612 INFO:teuthology.orchestra.run.smithi183.stdout:NAME="Red Hat Enterprise Linux" 2023-07-20T01:01:16.612 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION="8.6 (Ootpa)" 2023-07-20T01:01:16.612 INFO:teuthology.orchestra.run.smithi183.stdout:ID="rhel" 2023-07-20T01:01:16.612 INFO:teuthology.orchestra.run.smithi183.stdout:ID_LIKE="fedora" 2023-07-20T01:01:16.613 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_ID="8.6" 2023-07-20T01:01:16.613 INFO:teuthology.orchestra.run.smithi183.stdout:PLATFORM_ID="platform:el8" 2023-07-20T01:01:16.613 INFO:teuthology.orchestra.run.smithi183.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-20T01:01:16.613 INFO:teuthology.orchestra.run.smithi183.stdout:ANSI_COLOR="0;31" 2023-07-20T01:01:16.613 INFO:teuthology.orchestra.run.smithi183.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-20T01:01:16.613 INFO:teuthology.orchestra.run.smithi183.stdout:HOME_URL="https://www.redhat.com/" 2023-07-20T01:01:16.613 INFO:teuthology.orchestra.run.smithi183.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-20T01:01:16.613 INFO:teuthology.orchestra.run.smithi183.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-20T01:01:16.614 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-07-20T01:01:16.614 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-20T01:01:16.614 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-20T01:01:16.614 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-20T01:01:16.614 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-20T01:01:16.614 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2023-07-20T01:01:16.640 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-07-20T01:01:16.652 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-07-20T01:01:16.662 INFO:teuthology.task.internal:Checking for old test directory... 2023-07-20T01:01:16.663 DEBUG:teuthology.orchestra.run.smithi060:> test '!' -e /home/ubuntu/cephtest 2023-07-20T01:01:16.666 DEBUG:teuthology.orchestra.run.smithi136:> test '!' -e /home/ubuntu/cephtest 2023-07-20T01:01:16.668 DEBUG:teuthology.orchestra.run.smithi183:> test '!' -e /home/ubuntu/cephtest 2023-07-20T01:01:16.687 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-07-20T01:01:16.709 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-07-20T01:01:16.709 DEBUG:teuthology.orchestra.run.smithi060:> test -z $(ls -A /var/lib/ceph) 2023-07-20T01:01:16.726 DEBUG:teuthology.orchestra.run.smithi136:> test -z $(ls -A /var/lib/ceph) 2023-07-20T01:01:16.731 DEBUG:teuthology.orchestra.run.smithi183:> test -z $(ls -A /var/lib/ceph) 2023-07-20T01:01:16.779 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-07-20T01:01:16.866 INFO:teuthology.run_tasks:Running task kernel... 2023-07-20T01:01:16.885 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-07-20T01:01:16.886 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-07-20T01:01:16.887 DEBUG:teuthology.orchestra.run.smithi060:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-20T01:01:16.887 DEBUG:teuthology.orchestra.run.smithi136:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-20T01:01:16.888 DEBUG:teuthology.orchestra.run.smithi183:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-20T01:01:16.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T01:01:16.907 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2023-07-20T01:01:16.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T01:01:16.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T01:01:16.908 DEBUG:teuthology.orchestra.run.smithi060:> uname -r 2023-07-20T01:01:16.909 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2023-07-20T01:01:16.966 INFO:teuthology.orchestra.run.smithi183.stdout:4.18.0-372.9.1.el8.x86_64 2023-07-20T01:01:16.967 INFO:teuthology.task.kernel:Running kernel on smithi183: 4.18.0-372.9.1.el8.x86_64 2023-07-20T01:01:16.967 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum install -y kernel 2023-07-20T01:01:16.968 INFO:teuthology.orchestra.run.smithi060.stdout:4.18.0-372.9.1.el8.x86_64 2023-07-20T01:01:16.969 INFO:teuthology.task.kernel:Running kernel on smithi060: 4.18.0-372.9.1.el8.x86_64 2023-07-20T01:01:16.969 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum install -y kernel 2023-07-20T01:01:16.971 INFO:teuthology.orchestra.run.smithi136.stdout:4.18.0-372.9.1.el8.x86_64 2023-07-20T01:01:16.971 INFO:teuthology.task.kernel:Running kernel on smithi136: 4.18.0-372.9.1.el8.x86_64 2023-07-20T01:01:16.972 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum install -y kernel 2023-07-20T01:01:18.138 INFO:teuthology.orchestra.run.smithi183.stdout:Updating Subscription Management repositories. 2023-07-20T01:01:18.139 INFO:teuthology.orchestra.run.smithi183.stdout:Unable to read consumer identity 2023-07-20T01:01:18.191 INFO:teuthology.orchestra.run.smithi136.stdout:Updating Subscription Management repositories. 2023-07-20T01:01:18.191 INFO:teuthology.orchestra.run.smithi136.stdout:Unable to read consumer identity 2023-07-20T01:01:18.282 INFO:teuthology.orchestra.run.smithi060.stdout:Updating Subscription Management repositories. 2023-07-20T01:01:18.283 INFO:teuthology.orchestra.run.smithi060.stdout:Unable to read consumer identity 2023-07-20T01:01:19.674 INFO:teuthology.orchestra.run.smithi183.stdout:Copr repo for python3-asyncssh owned by ceph 9.8 kB/s | 3.5 kB 00:00 2023-07-20T01:01:19.701 INFO:teuthology.orchestra.run.smithi136.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2023-07-20T01:01:19.731 INFO:teuthology.orchestra.run.smithi060.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2023-07-20T01:01:21.184 INFO:teuthology.orchestra.run.smithi136.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2023-07-20T01:01:21.458 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 8.9 MB/s | 16 MB 00:01 2023-07-20T01:01:22.681 INFO:teuthology.orchestra.run.smithi060.stdout:Extra Packages for Enterprise Linux 5.3 MB/s | 16 MB 00:02 2023-07-20T01:01:24.651 INFO:teuthology.orchestra.run.smithi136.stdout:lab-extras 535 kB/s | 24 kB 00:00 2023-07-20T01:01:24.899 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 430 kB/s | 24 kB 00:00 2023-07-20T01:01:25.278 INFO:teuthology.orchestra.run.smithi136.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-07-20T01:01:25.326 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2023-07-20T01:01:25.326 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2023-07-20T01:01:25.326 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2023-07-20T01:01:25.468 DEBUG:teuthology.orchestra.run.smithi136:> echo no | sudo yum reinstall kernel || true 2023-07-20T01:01:25.577 INFO:teuthology.orchestra.run.smithi183.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-07-20T01:01:25.626 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-07-20T01:01:25.627 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2023-07-20T01:01:25.628 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-07-20T01:01:25.737 DEBUG:teuthology.orchestra.run.smithi183:> echo no | sudo yum reinstall kernel || true 2023-07-20T01:01:25.837 INFO:teuthology.orchestra.run.smithi136.stdout:Updating Subscription Management repositories. 2023-07-20T01:01:25.837 INFO:teuthology.orchestra.run.smithi136.stdout:Unable to read consumer identity 2023-07-20T01:01:26.068 INFO:teuthology.orchestra.run.smithi183.stdout:Updating Subscription Management repositories. 2023-07-20T01:01:26.068 INFO:teuthology.orchestra.run.smithi183.stdout:Unable to read consumer identity 2023-07-20T01:01:26.101 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:02 ago on Thu 20 Jul 2023 01:01:24 AM UTC. 2023-07-20T01:01:26.140 INFO:teuthology.orchestra.run.smithi136.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-20T01:01:26.184 INFO:teuthology.orchestra.run.smithi136.stderr:Error: No packages marked for reinstall. 2023-07-20T01:01:26.232 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum reinstall -y kernel || true 2023-07-20T01:01:26.274 INFO:teuthology.orchestra.run.smithi060.stdout:lab-extras 423 kB/s | 24 kB 00:00 2023-07-20T01:01:26.335 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:02 ago on Thu 20 Jul 2023 01:01:24 AM UTC. 2023-07-20T01:01:26.375 INFO:teuthology.orchestra.run.smithi183.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-20T01:01:26.418 INFO:teuthology.orchestra.run.smithi183.stderr:Error: No packages marked for reinstall. 2023-07-20T01:01:26.465 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum reinstall -y kernel || true 2023-07-20T01:01:26.560 INFO:teuthology.orchestra.run.smithi136.stdout:Updating Subscription Management repositories. 2023-07-20T01:01:26.561 INFO:teuthology.orchestra.run.smithi136.stdout:Unable to read consumer identity 2023-07-20T01:01:26.796 INFO:teuthology.orchestra.run.smithi183.stdout:Updating Subscription Management repositories. 2023-07-20T01:01:26.796 INFO:teuthology.orchestra.run.smithi183.stdout:Unable to read consumer identity 2023-07-20T01:01:26.816 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:02 ago on Thu 20 Jul 2023 01:01:24 AM UTC. 2023-07-20T01:01:26.856 INFO:teuthology.orchestra.run.smithi136.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-20T01:01:26.888 INFO:teuthology.orchestra.run.smithi136.stderr:Error: No packages marked for reinstall. 2023-07-20T01:01:26.935 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q kernel | sort -rV | head -n 1 2023-07-20T01:01:26.975 INFO:teuthology.orchestra.run.smithi060.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-07-20T01:01:27.016 INFO:teuthology.orchestra.run.smithi136.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-07-20T01:01:27.016 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-07-20T01:01:27.017 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-07-20T01:01:27.017 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-07-20T01:01:27.017 DEBUG:teuthology.orchestra.run.smithi136:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-07-20T01:01:27.022 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2023-07-20T01:01:27.023 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2023-07-20T01:01:27.023 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2023-07-20T01:01:27.057 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:03 ago on Thu 20 Jul 2023 01:01:24 AM UTC. 2023-07-20T01:01:27.074 INFO:teuthology.orchestra.run.smithi136.stdout:ttyS1 2023-07-20T01:01:27.087 DEBUG:teuthology.parallel:result is None 2023-07-20T01:01:27.096 INFO:teuthology.orchestra.run.smithi183.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-20T01:01:27.120 INFO:teuthology.orchestra.run.smithi183.stderr:Error: No packages marked for reinstall. 2023-07-20T01:01:27.167 DEBUG:teuthology.orchestra.run.smithi060:> echo no | sudo yum reinstall kernel || true 2023-07-20T01:01:27.169 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q kernel | sort -rV | head -n 1 2023-07-20T01:01:27.249 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-07-20T01:01:27.250 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-07-20T01:01:27.251 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-07-20T01:01:27.251 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2023-07-20T01:01:27.251 DEBUG:teuthology.orchestra.run.smithi183:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-07-20T01:01:27.283 INFO:teuthology.orchestra.run.smithi183.stdout:ttyS1 2023-07-20T01:01:27.303 DEBUG:teuthology.parallel:result is None 2023-07-20T01:01:27.516 INFO:teuthology.orchestra.run.smithi060.stdout:Updating Subscription Management repositories. 2023-07-20T01:01:27.517 INFO:teuthology.orchestra.run.smithi060.stdout:Unable to read consumer identity 2023-07-20T01:01:27.790 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:00:01 ago on Thu 20 Jul 2023 01:01:26 AM UTC. 2023-07-20T01:01:27.835 INFO:teuthology.orchestra.run.smithi060.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-20T01:01:27.874 INFO:teuthology.orchestra.run.smithi060.stderr:Error: No packages marked for reinstall. 2023-07-20T01:01:27.921 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum reinstall -y kernel || true 2023-07-20T01:01:28.270 INFO:teuthology.orchestra.run.smithi060.stdout:Updating Subscription Management repositories. 2023-07-20T01:01:28.271 INFO:teuthology.orchestra.run.smithi060.stdout:Unable to read consumer identity 2023-07-20T01:01:28.540 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:00:02 ago on Thu 20 Jul 2023 01:01:26 AM UTC. 2023-07-20T01:01:28.581 INFO:teuthology.orchestra.run.smithi060.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-20T01:01:28.615 INFO:teuthology.orchestra.run.smithi060.stderr:Error: No packages marked for reinstall. 2023-07-20T01:01:28.661 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q kernel | sort -rV | head -n 1 2023-07-20T01:01:28.744 INFO:teuthology.orchestra.run.smithi060.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-07-20T01:01:28.745 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-07-20T01:01:28.745 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-07-20T01:01:28.745 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-07-20T01:01:28.746 DEBUG:teuthology.orchestra.run.smithi060:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-07-20T01:01:28.779 INFO:teuthology.orchestra.run.smithi060.stdout:ttyS1 2023-07-20T01:01:28.791 DEBUG:teuthology.parallel:result is None 2023-07-20T01:01:28.792 INFO:teuthology.run_tasks:Running task internal.base... 2023-07-20T01:01:28.799 INFO:teuthology.task.internal:Creating test directory... 2023-07-20T01:01:28.800 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-20T01:01:28.834 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-20T01:01:28.837 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-20T01:01:28.856 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-07-20T01:01:28.868 INFO:teuthology.run_tasks:Running task internal.archive... 2023-07-20T01:01:28.883 INFO:teuthology.task.internal:Creating archive directory... 2023-07-20T01:01:28.883 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-20T01:01:28.894 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-20T01:01:28.898 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-20T01:01:28.925 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-07-20T01:01:28.938 INFO:teuthology.task.internal:Enabling coredump saving... 2023-07-20T01:01:28.938 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-20T01:01:28.956 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-20T01:01:28.958 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-20T01:01:28.984 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T01:01:28.992 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T01:01:28.995 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T01:01:29.004 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T01:01:29.005 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T01:01:29.017 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-20T01:01:29.019 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-07-20T01:01:29.027 INFO:teuthology.task.internal:Configuring sudo... 2023-07-20T01:01:29.027 DEBUG:teuthology.orchestra.run.smithi060:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-20T01:01:29.040 DEBUG:teuthology.orchestra.run.smithi136:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-20T01:01:29.043 DEBUG:teuthology.orchestra.run.smithi183:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-20T01:01:29.074 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-07-20T01:01:29.087 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-07-20T01:01:29.087 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-20T01:01:29.108 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-20T01:01:29.112 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-20T01:01:29.132 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T01:01:29.210 DEBUG:teuthology.orchestra.run.smithi060:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T01:01:29.255 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T01:01:29.341 DEBUG:teuthology.orchestra.run.smithi060:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T01:01:29.381 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-07-20T01:01:29.382 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-20T01:01:29.448 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T01:01:29.497 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T01:01:29.572 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T01:01:29.620 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T01:01:29.662 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-07-20T01:01:29.662 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-20T01:01:29.731 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T01:01:29.780 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-20T01:01:29.835 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T01:01:29.926 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-20T01:01:29.966 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2023-07-20T01:01:29.966 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-20T01:01:30.034 DEBUG:teuthology.orchestra.run.smithi060:> sudo service rsyslog restart 2023-07-20T01:01:30.037 DEBUG:teuthology.orchestra.run.smithi136:> sudo service rsyslog restart 2023-07-20T01:01:30.039 DEBUG:teuthology.orchestra.run.smithi183:> sudo service rsyslog restart 2023-07-20T01:01:30.095 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T01:01:30.110 INFO:teuthology.orchestra.run.smithi060.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T01:01:30.129 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T01:01:30.483 INFO:teuthology.run_tasks:Running task internal.timer... 2023-07-20T01:01:30.492 INFO:teuthology.task.internal:Starting timer... 2023-07-20T01:01:30.492 INFO:teuthology.run_tasks:Running task pcp... 2023-07-20T01:01:30.508 INFO:teuthology.run_tasks:Running task selinux... 2023-07-20T01:01:30.523 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-07-20T01:01:30.524 DEBUG:teuthology.orchestra.run.smithi060:> sudo service auditd rotate 2023-07-20T01:01:30.584 INFO:teuthology.orchestra.run.smithi060.stdout:Rotating logs: 2023-07-20T01:01:30.585 DEBUG:teuthology.orchestra.run.smithi136:> sudo service auditd rotate 2023-07-20T01:01:30.637 INFO:teuthology.orchestra.run.smithi136.stdout:Rotating logs: 2023-07-20T01:01:30.639 DEBUG:teuthology.orchestra.run.smithi183:> sudo service auditd rotate 2023-07-20T01:01:30.706 INFO:teuthology.orchestra.run.smithi183.stdout:Rotating logs: 2023-07-20T01:01:30.708 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-07-20T01:01:30.708 DEBUG:teuthology.orchestra.run.smithi060:> /usr/sbin/getenforce 2023-07-20T01:01:30.737 INFO:teuthology.orchestra.run.smithi060.stdout:Permissive 2023-07-20T01:01:30.737 DEBUG:teuthology.orchestra.run.smithi136:> /usr/sbin/getenforce 2023-07-20T01:01:30.766 INFO:teuthology.orchestra.run.smithi136.stdout:Permissive 2023-07-20T01:01:30.767 DEBUG:teuthology.orchestra.run.smithi183:> /usr/sbin/getenforce 2023-07-20T01:01:30.785 INFO:teuthology.orchestra.run.smithi183.stdout:Permissive 2023-07-20T01:01:30.785 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi060.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi136.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi183.front.sepia.ceph.com': 'permissive'} 2023-07-20T01:01:30.785 DEBUG:teuthology.orchestra.run.smithi060:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T01:01:30.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T01:01:30.813 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T01:01:30.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T01:01:30.843 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T01:01:30.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T01:01:30.872 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-07-20T01:01:30.872 DEBUG:teuthology.orchestra.run.smithi060:> sudo /usr/sbin/setenforce permissive 2023-07-20T01:01:30.901 DEBUG:teuthology.orchestra.run.smithi136:> sudo /usr/sbin/setenforce permissive 2023-07-20T01:01:30.928 DEBUG:teuthology.orchestra.run.smithi183:> sudo /usr/sbin/setenforce permissive 2023-07-20T01:01:30.956 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-07-20T01:01:30.970 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-07-20T01:01:30.976 INFO:teuthology.repo_utils:Fetching main from origin 2023-07-20T01:01:31.150 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-07-20T01:01:31.170 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-07-20T01:01:31.171 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi060.front.sepia.ceph.com,smithi136.front.sepia.ceph.com,smithi183.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-07-20T01:11:15.677 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi060.front.sepia.ceph.com'), Remote(name='ubuntu@smithi136.front.sepia.ceph.com'), Remote(name='ubuntu@smithi183.front.sepia.ceph.com')] 2023-07-20T01:11:15.679 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2023-07-20T01:11:15.680 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T01:11:15.749 DEBUG:teuthology.orchestra.run.smithi060:> true 2023-07-20T01:11:15.817 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi060.front.sepia.ceph.com' 2023-07-20T01:11:15.818 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2023-07-20T01:11:15.818 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T01:11:15.886 DEBUG:teuthology.orchestra.run.smithi136:> true 2023-07-20T01:11:15.953 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2023-07-20T01:11:15.953 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2023-07-20T01:11:15.954 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-20T01:11:16.022 DEBUG:teuthology.orchestra.run.smithi183:> true 2023-07-20T01:11:16.095 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2023-07-20T01:11:16.096 INFO:teuthology.run_tasks:Running task clock... 2023-07-20T01:11:16.107 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-07-20T01:11:16.108 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-20T01:11:16.108 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T01:11:16.110 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-20T01:11:16.111 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T01:11:16.113 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-20T01:11:16.113 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T01:11:16.143 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-20T01:11:16.144 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-20T01:11:16.159 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-20T01:11:16.160 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-20T01:11:16.171 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-20T01:11:16.187 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-20T01:11:16.236 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: ntpd: command not found 2023-07-20T01:11:16.242 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: ntpd: command not found 2023-07-20T01:11:16.248 INFO:teuthology.orchestra.run.smithi060.stdout:506 Cannot talk to daemon 2023-07-20T01:11:16.254 INFO:teuthology.orchestra.run.smithi136.stdout:506 Cannot talk to daemon 2023-07-20T01:11:16.256 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: ntpd: command not found 2023-07-20T01:11:16.264 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-20T01:11:16.268 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-20T01:11:16.269 INFO:teuthology.orchestra.run.smithi183.stdout:506 Cannot talk to daemon 2023-07-20T01:11:16.278 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-20T01:11:16.281 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-20T01:11:16.283 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-20T01:11:16.297 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-20T01:11:16.350 INFO:teuthology.orchestra.run.smithi136.stderr:bash: ntpq: command not found 2023-07-20T01:11:16.353 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T01:11:16.354 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2023-07-20T01:11:16.354 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T01:11:16.354 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T01:11:16.354 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T01:11:16.354 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T01:11:16.358 INFO:teuthology.orchestra.run.smithi060.stderr:bash: ntpq: command not found 2023-07-20T01:11:16.361 INFO:teuthology.orchestra.run.smithi060.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T01:11:16.362 INFO:teuthology.orchestra.run.smithi060.stdout:=============================================================================== 2023-07-20T01:11:16.362 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T01:11:16.362 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T01:11:16.362 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T01:11:16.362 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T01:11:16.368 INFO:teuthology.orchestra.run.smithi183.stderr:bash: ntpq: command not found 2023-07-20T01:11:16.373 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T01:11:16.373 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2023-07-20T01:11:16.373 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T01:11:16.373 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T01:11:16.373 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T01:11:16.374 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T01:11:16.374 INFO:teuthology.run_tasks:Running task pexec... 2023-07-20T01:11:16.384 INFO:teuthology.task.pexec:Executing custom commands... 2023-07-20T01:11:16.384 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi060.front.sepia.ceph.com 2023-07-20T01:11:16.385 DEBUG:teuthology.orchestra.run.smithi060:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-20T01:11:16.385 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi136.front.sepia.ceph.com 2023-07-20T01:11:16.386 DEBUG:teuthology.orchestra.run.smithi136:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-20T01:11:16.386 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi183.front.sepia.ceph.com 2023-07-20T01:11:16.386 DEBUG:teuthology.orchestra.run.smithi183:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-20T01:11:16.767 INFO:teuthology.orchestra.run.smithi136.stdout:Updating Subscription Management repositories. 2023-07-20T01:11:16.785 INFO:teuthology.orchestra.run.smithi183.stdout:Updating Subscription Management repositories. 2023-07-20T01:11:16.794 INFO:teuthology.orchestra.run.smithi060.stdout:Updating Subscription Management repositories. 2023-07-20T01:11:18.367 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:02:10 ago on Thu 20 Jul 2023 01:09:07 AM UTC. 2023-07-20T01:11:18.400 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:00 ago on Thu 20 Jul 2023 01:09:17 AM UTC. 2023-07-20T01:11:18.420 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:39 ago on Thu 20 Jul 2023 01:09:38 AM UTC. 2023-07-20T01:11:20.105 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2023-07-20T01:11:20.106 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2023-07-20T01:11:20.106 INFO:teuthology.orchestra.run.smithi136.stdout: Package Architecture Version Repository Size 2023-07-20T01:11:20.106 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2023-07-20T01:11:20.106 INFO:teuthology.orchestra.run.smithi136.stdout:Resetting modules: 2023-07-20T01:11:20.106 INFO:teuthology.orchestra.run.smithi136.stdout: container-tools 2023-07-20T01:11:20.106 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-07-20T01:11:20.107 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2023-07-20T01:11:20.107 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2023-07-20T01:11:20.107 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-07-20T01:11:20.116 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-07-20T01:11:20.117 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-07-20T01:11:20.117 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2023-07-20T01:11:20.117 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-07-20T01:11:20.117 INFO:teuthology.orchestra.run.smithi183.stdout:Resetting modules: 2023-07-20T01:11:20.117 INFO:teuthology.orchestra.run.smithi183.stdout: container-tools 2023-07-20T01:11:20.117 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-07-20T01:11:20.118 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-07-20T01:11:20.118 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-07-20T01:11:20.118 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-07-20T01:11:20.416 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2023-07-20T01:11:20.417 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-07-20T01:11:20.417 INFO:teuthology.orchestra.run.smithi060.stdout: Package Architecture Version Repository Size 2023-07-20T01:11:20.417 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-07-20T01:11:20.417 INFO:teuthology.orchestra.run.smithi060.stdout:Resetting modules: 2023-07-20T01:11:20.417 INFO:teuthology.orchestra.run.smithi060.stdout: container-tools 2023-07-20T01:11:20.418 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:11:20.418 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2023-07-20T01:11:20.418 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2023-07-20T01:11:20.418 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:11:21.060 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:02:13 ago on Thu 20 Jul 2023 01:09:07 AM UTC. 2023-07-20T01:11:21.212 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:03 ago on Thu 20 Jul 2023 01:09:17 AM UTC. 2023-07-20T01:11:21.414 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:42 ago on Thu 20 Jul 2023 01:09:38 AM UTC. 2023-07-20T01:11:27.699 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2023-07-20T01:11:27.800 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-07-20T01:11:28.154 INFO:teuthology.orchestra.run.smithi136.stdout:Updating Subscription Management repositories. 2023-07-20T01:11:28.450 INFO:teuthology.orchestra.run.smithi183.stdout:Updating Subscription Management repositories. 2023-07-20T01:11:28.768 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2023-07-20T01:11:29.253 INFO:teuthology.orchestra.run.smithi060.stdout:Updating Subscription Management repositories. 2023-07-20T01:11:29.516 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:02:22 ago on Thu 20 Jul 2023 01:09:07 AM UTC. 2023-07-20T01:11:29.836 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:12 ago on Thu 20 Jul 2023 01:09:17 AM UTC. 2023-07-20T01:11:30.316 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:51 ago on Thu 20 Jul 2023 01:09:38 AM UTC. 2023-07-20T01:11:31.254 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2023-07-20T01:11:31.349 INFO:teuthology.orchestra.run.smithi136.stdout:============================================================================================================ 2023-07-20T01:11:31.350 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2023-07-20T01:11:31.350 INFO:teuthology.orchestra.run.smithi136.stdout:============================================================================================================ 2023-07-20T01:11:31.350 INFO:teuthology.orchestra.run.smithi136.stdout:Installing group/module packages: 2023-07-20T01:11:31.350 INFO:teuthology.orchestra.run.smithi136.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-20T01:11:31.350 INFO:teuthology.orchestra.run.smithi136.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-20T01:11:31.351 INFO:teuthology.orchestra.run.smithi136.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-20T01:11:31.351 INFO:teuthology.orchestra.run.smithi136.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-20T01:11:31.351 INFO:teuthology.orchestra.run.smithi136.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-20T01:11:31.351 INFO:teuthology.orchestra.run.smithi136.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-20T01:11:31.351 INFO:teuthology.orchestra.run.smithi136.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-20T01:11:31.351 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2023-07-20T01:11:31.351 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-20T01:11:31.352 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-20T01:11:31.352 INFO:teuthology.orchestra.run.smithi136.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-20T01:11:31.352 INFO:teuthology.orchestra.run.smithi136.stdout:Installing module profiles: 2023-07-20T01:11:31.352 INFO:teuthology.orchestra.run.smithi136.stdout: container-tools/common 2023-07-20T01:11:31.352 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-07-20T01:11:31.352 INFO:teuthology.orchestra.run.smithi136.stdout:Enabling module streams: 2023-07-20T01:11:31.352 INFO:teuthology.orchestra.run.smithi136.stdout: container-tools rhel8 2023-07-20T01:11:31.353 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-07-20T01:11:31.353 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2023-07-20T01:11:31.353 INFO:teuthology.orchestra.run.smithi136.stdout:============================================================================================================ 2023-07-20T01:11:31.353 INFO:teuthology.orchestra.run.smithi136.stdout:Install 10 Packages 2023-07-20T01:11:31.353 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-07-20T01:11:31.354 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 18 M 2023-07-20T01:11:31.354 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 65 M 2023-07-20T01:11:31.354 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2023-07-20T01:11:31.471 INFO:teuthology.orchestra.run.smithi136.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 790 kB/s | 94 kB 00:00 2023-07-20T01:11:31.521 INFO:teuthology.orchestra.run.smithi136.stdout:(2/10): yajl-2.1.0-10.el8.x86_64.rpm 241 kB/s | 41 kB 00:00 2023-07-20T01:11:31.538 INFO:teuthology.orchestra.run.smithi136.stdout:(3/10): python3-pytoml-0.1.14-5.git7dea353.el8. 133 kB/s | 25 kB 00:00 2023-07-20T01:11:31.564 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-07-20T01:11:31.619 INFO:teuthology.orchestra.run.smithi136.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 333 kB/s | 49 kB 00:00 2023-07-20T01:11:31.655 INFO:teuthology.orchestra.run.smithi136.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 3.8 MB/s | 524 kB 00:00 2023-07-20T01:11:31.659 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================================ 2023-07-20T01:11:31.659 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2023-07-20T01:11:31.659 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================================ 2023-07-20T01:11:31.659 INFO:teuthology.orchestra.run.smithi183.stdout:Installing group/module packages: 2023-07-20T01:11:31.659 INFO:teuthology.orchestra.run.smithi183.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-20T01:11:31.659 INFO:teuthology.orchestra.run.smithi183.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-20T01:11:31.660 INFO:teuthology.orchestra.run.smithi183.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-20T01:11:31.660 INFO:teuthology.orchestra.run.smithi183.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-20T01:11:31.660 INFO:teuthology.orchestra.run.smithi183.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-20T01:11:31.660 INFO:teuthology.orchestra.run.smithi183.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-20T01:11:31.660 INFO:teuthology.orchestra.run.smithi183.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-20T01:11:31.660 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2023-07-20T01:11:31.660 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-20T01:11:31.661 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-20T01:11:31.661 INFO:teuthology.orchestra.run.smithi183.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-20T01:11:31.661 INFO:teuthology.orchestra.run.smithi183.stdout:Installing module profiles: 2023-07-20T01:11:31.661 INFO:teuthology.orchestra.run.smithi183.stdout: container-tools/common 2023-07-20T01:11:31.661 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-07-20T01:11:31.661 INFO:teuthology.orchestra.run.smithi183.stdout:Enabling module streams: 2023-07-20T01:11:31.661 INFO:teuthology.orchestra.run.smithi183.stdout: container-tools rhel8 2023-07-20T01:11:31.662 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-07-20T01:11:31.662 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-07-20T01:11:31.662 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================================ 2023-07-20T01:11:31.662 INFO:teuthology.orchestra.run.smithi183.stdout:Install 10 Packages 2023-07-20T01:11:31.662 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-07-20T01:11:31.663 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 18 M 2023-07-20T01:11:31.663 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 65 M 2023-07-20T01:11:31.663 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-07-20T01:11:31.818 INFO:teuthology.orchestra.run.smithi183.stdout:(1/10): yajl-2.1.0-10.el8.x86_64.rpm 261 kB/s | 41 kB 00:00 2023-07-20T01:11:31.822 INFO:teuthology.orchestra.run.smithi136.stdout:(6/10): buildah-1.26.2-1.module+el8.6.0+16771+2 28 MB/s | 8.0 MB 00:00 2023-07-20T01:11:31.843 INFO:teuthology.orchestra.run.smithi183.stdout:(2/10): python3-pyxdg-0.25-16.el8.noarch.rpm 517 kB/s | 94 kB 00:00 2023-07-20T01:11:31.856 INFO:teuthology.orchestra.run.smithi136.stdout:(7/10): python3-podman-4.0.1-1.module+el8.6.0+1 633 kB/s | 149 kB 00:00 2023-07-20T01:11:31.881 INFO:teuthology.orchestra.run.smithi136.stdout:(8/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 943 kB/s | 213 kB 00:00 2023-07-20T01:11:31.918 INFO:teuthology.orchestra.run.smithi183.stdout:(3/10): python3-pytoml-0.1.14-5.git7dea353.el8. 96 kB/s | 25 kB 00:00 2023-07-20T01:11:31.952 INFO:teuthology.orchestra.run.smithi183.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 369 kB/s | 49 kB 00:00 2023-07-20T01:11:31.973 INFO:teuthology.orchestra.run.smithi136.stdout:(9/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 19 MB/s | 2.2 MB 00:00 2023-07-20T01:11:31.994 INFO:teuthology.orchestra.run.smithi183.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 3.4 MB/s | 524 kB 00:00 2023-07-20T01:11:32.053 INFO:teuthology.orchestra.run.smithi183.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 1.5 MB/s | 149 kB 00:00 2023-07-20T01:11:32.086 INFO:teuthology.orchestra.run.smithi183.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 2.3 MB/s | 213 kB 00:00 2023-07-20T01:11:32.140 INFO:teuthology.orchestra.run.smithi136.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 22 MB/s | 7.0 MB 00:00 2023-07-20T01:11:32.140 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2023-07-20T01:11:32.141 INFO:teuthology.orchestra.run.smithi136.stdout:Total 23 MB/s | 18 MB 00:00 2023-07-20T01:11:32.228 INFO:teuthology.orchestra.run.smithi183.stdout:(8/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 15 MB/s | 2.2 MB 00:00 2023-07-20T01:11:32.289 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2023-07-20T01:11:32.340 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2023-07-20T01:11:32.359 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2023-07-20T01:11:32.359 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2023-07-20T01:11:32.362 INFO:teuthology.orchestra.run.smithi183.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 23 MB/s | 7.0 MB 00:00 2023-07-20T01:11:32.389 INFO:teuthology.orchestra.run.smithi060.stdout:============================================================================================================ 2023-07-20T01:11:32.389 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2023-07-20T01:11:32.389 INFO:teuthology.orchestra.run.smithi060.stdout:============================================================================================================ 2023-07-20T01:11:32.389 INFO:teuthology.orchestra.run.smithi060.stdout:Installing group/module packages: 2023-07-20T01:11:32.390 INFO:teuthology.orchestra.run.smithi060.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-20T01:11:32.390 INFO:teuthology.orchestra.run.smithi060.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-20T01:11:32.390 INFO:teuthology.orchestra.run.smithi060.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-20T01:11:32.390 INFO:teuthology.orchestra.run.smithi060.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-20T01:11:32.390 INFO:teuthology.orchestra.run.smithi060.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-20T01:11:32.390 INFO:teuthology.orchestra.run.smithi060.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-20T01:11:32.391 INFO:teuthology.orchestra.run.smithi060.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-20T01:11:32.391 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2023-07-20T01:11:32.391 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-20T01:11:32.391 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-20T01:11:32.391 INFO:teuthology.orchestra.run.smithi060.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-20T01:11:32.391 INFO:teuthology.orchestra.run.smithi060.stdout:Installing module profiles: 2023-07-20T01:11:32.391 INFO:teuthology.orchestra.run.smithi060.stdout: container-tools/common 2023-07-20T01:11:32.392 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:11:32.392 INFO:teuthology.orchestra.run.smithi060.stdout:Enabling module streams: 2023-07-20T01:11:32.392 INFO:teuthology.orchestra.run.smithi060.stdout: container-tools rhel8 2023-07-20T01:11:32.392 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:11:32.392 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2023-07-20T01:11:32.392 INFO:teuthology.orchestra.run.smithi060.stdout:============================================================================================================ 2023-07-20T01:11:32.392 INFO:teuthology.orchestra.run.smithi060.stdout:Install 10 Packages 2023-07-20T01:11:32.393 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:11:32.393 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 18 M 2023-07-20T01:11:32.393 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 65 M 2023-07-20T01:11:32.393 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2023-07-20T01:11:32.462 INFO:teuthology.orchestra.run.smithi183.stdout:(10/10): buildah-1.26.2-1.module+el8.6.0+16771+ 15 MB/s | 8.0 MB 00:00 2023-07-20T01:11:32.462 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-07-20T01:11:32.463 INFO:teuthology.orchestra.run.smithi183.stdout:Total 23 MB/s | 18 MB 00:00 2023-07-20T01:11:32.541 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2023-07-20T01:11:32.545 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2023-07-20T01:11:32.558 INFO:teuthology.orchestra.run.smithi060.stdout:(1/10): yajl-2.1.0-10.el8.x86_64.rpm 246 kB/s | 41 kB 00:00 2023-07-20T01:11:32.583 INFO:teuthology.orchestra.run.smithi060.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 130 kB/s | 25 kB 00:00 2023-07-20T01:11:32.609 INFO:teuthology.orchestra.run.smithi060.stdout:(3/10): python3-pyxdg-0.25-16.el8.noarch.rpm 433 kB/s | 94 kB 00:00 2023-07-20T01:11:32.663 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-07-20T01:11:32.676 INFO:teuthology.orchestra.run.smithi060.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 421 kB/s | 49 kB 00:00 2023-07-20T01:11:32.681 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-07-20T01:11:32.682 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-07-20T01:11:32.717 INFO:teuthology.orchestra.run.smithi060.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 3.8 MB/s | 524 kB 00:00 2023-07-20T01:11:32.809 INFO:teuthology.orchestra.run.smithi060.stdout:(6/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 2.3 MB/s | 213 kB 00:00 2023-07-20T01:11:32.826 INFO:teuthology.orchestra.run.smithi060.stdout:(7/10): python3-podman-4.0.1-1.module+el8.6.0+1 993 kB/s | 149 kB 00:00 2023-07-20T01:11:32.862 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-07-20T01:11:32.866 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-07-20T01:11:32.935 INFO:teuthology.orchestra.run.smithi060.stdout:(8/10): buildah-1.26.2-1.module+el8.6.0+16771+2 24 MB/s | 8.0 MB 00:00 2023-07-20T01:11:33.027 INFO:teuthology.orchestra.run.smithi060.stdout:(9/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 11 MB/s | 2.2 MB 00:00 2023-07-20T01:11:33.063 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2023-07-20T01:11:33.152 INFO:teuthology.orchestra.run.smithi060.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 20 MB/s | 7.0 MB 00:00 2023-07-20T01:11:33.153 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2023-07-20T01:11:33.153 INFO:teuthology.orchestra.run.smithi060.stdout:Total 24 MB/s | 18 MB 00:00 2023-07-20T01:11:33.277 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-20T01:11:33.347 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-07-20T01:11:33.359 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2023-07-20T01:11:33.379 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2023-07-20T01:11:33.379 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2023-07-20T01:11:33.462 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T01:11:33.529 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-20T01:11:33.573 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2023-07-20T01:11:33.577 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2023-07-20T01:11:33.694 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T01:11:33.698 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-20T01:11:33.888 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-20T01:11:33.907 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-20T01:11:34.085 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2023-07-20T01:11:34.101 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-20T01:11:34.276 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-20T01:11:34.317 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-20T01:11:34.428 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T01:11:34.448 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-20T01:11:34.634 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-20T01:11:34.831 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-20T01:11:35.120 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-20T01:11:35.211 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-20T01:11:35.220 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-20T01:11:36.020 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-20T01:11:36.042 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-20T01:11:36.109 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-20T01:11:36.244 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-20T01:11:36.335 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-20T01:11:36.452 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-20T01:11:36.526 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-20T01:11:36.638 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T01:11:36.703 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T01:11:36.894 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T01:11:36.895 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-20T01:11:36.895 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T01:11:36.895 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-20T01:11:36.896 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-20T01:11:36.896 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-20T01:11:36.896 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-20T01:11:36.896 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-20T01:11:36.897 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-20T01:11:36.897 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-20T01:11:36.966 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-20T01:11:36.974 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T01:11:36.974 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-20T01:11:36.974 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T01:11:36.975 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-20T01:11:36.975 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-20T01:11:36.975 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-20T01:11:36.975 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-20T01:11:36.975 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-20T01:11:36.975 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-20T01:11:36.975 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-20T01:11:37.201 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-20T01:11:37.378 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-20T01:11:37.536 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T01:11:37.802 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-20T01:11:37.802 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-20T01:11:37.803 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-20T01:11:37.803 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-20T01:11:37.803 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-20T01:11:37.804 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-20T01:11:37.804 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-20T01:11:37.804 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-20T01:11:37.804 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-20T01:11:37.805 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-20T01:11:38.017 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-20T01:11:38.017 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:02:30 ago on Thu 20 Jul 2023 01:09:07 AM UTC. 2023-07-20T01:11:38.094 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-20T01:11:38.094 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:20 ago on Thu 20 Jul 2023 01:09:17 AM UTC. 2023-07-20T01:11:38.971 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-20T01:11:38.972 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:02:00 ago on Thu 20 Jul 2023 01:09:38 AM UTC. 2023-07-20T01:11:44.248 INFO:teuthology.orchestra.run.smithi136.stdout:Installed products updated. 2023-07-20T01:11:44.446 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-07-20T01:11:44.447 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2023-07-20T01:11:44.447 INFO:teuthology.orchestra.run.smithi136.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T01:11:44.447 INFO:teuthology.orchestra.run.smithi136.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T01:11:44.447 INFO:teuthology.orchestra.run.smithi136.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T01:11:44.447 INFO:teuthology.orchestra.run.smithi136.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-20T01:11:44.447 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-20T01:11:44.447 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-20T01:11:44.448 INFO:teuthology.orchestra.run.smithi136.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T01:11:44.448 INFO:teuthology.orchestra.run.smithi136.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T01:11:44.448 INFO:teuthology.orchestra.run.smithi136.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T01:11:44.448 INFO:teuthology.orchestra.run.smithi136.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-20T01:11:44.448 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-07-20T01:11:44.448 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2023-07-20T01:11:44.557 INFO:teuthology.orchestra.run.smithi183.stdout:Installed products updated. 2023-07-20T01:11:44.604 DEBUG:teuthology.parallel:result is None 2023-07-20T01:11:44.758 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-07-20T01:11:44.758 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-07-20T01:11:44.759 INFO:teuthology.orchestra.run.smithi183.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T01:11:44.759 INFO:teuthology.orchestra.run.smithi183.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T01:11:44.759 INFO:teuthology.orchestra.run.smithi183.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T01:11:44.759 INFO:teuthology.orchestra.run.smithi183.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-20T01:11:44.759 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-20T01:11:44.759 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-20T01:11:44.759 INFO:teuthology.orchestra.run.smithi183.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T01:11:44.760 INFO:teuthology.orchestra.run.smithi183.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T01:11:44.760 INFO:teuthology.orchestra.run.smithi183.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T01:11:44.760 INFO:teuthology.orchestra.run.smithi183.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-20T01:11:44.760 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-07-20T01:11:44.760 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-07-20T01:11:44.911 DEBUG:teuthology.parallel:result is None 2023-07-20T01:11:46.600 INFO:teuthology.orchestra.run.smithi060.stdout:Installed products updated. 2023-07-20T01:11:46.804 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:11:46.804 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2023-07-20T01:11:46.804 INFO:teuthology.orchestra.run.smithi060.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T01:11:46.804 INFO:teuthology.orchestra.run.smithi060.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T01:11:46.805 INFO:teuthology.orchestra.run.smithi060.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T01:11:46.805 INFO:teuthology.orchestra.run.smithi060.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-20T01:11:46.805 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-20T01:11:46.805 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-20T01:11:46.805 INFO:teuthology.orchestra.run.smithi060.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T01:11:46.805 INFO:teuthology.orchestra.run.smithi060.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-20T01:11:46.805 INFO:teuthology.orchestra.run.smithi060.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-20T01:11:46.805 INFO:teuthology.orchestra.run.smithi060.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-20T01:11:46.806 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:11:46.806 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2023-07-20T01:11:46.971 DEBUG:teuthology.parallel:result is None 2023-07-20T01:11:46.972 INFO:teuthology.run_tasks:Running task install... 2023-07-20T01:11:46.980 DEBUG:teuthology.task.install:project ceph 2023-07-20T01:11:46.981 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537'}} 2023-07-20T01:11:46.981 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537'} 2023-07-20T01:11:46.981 INFO:teuthology.task.install:Using flavor: default 2023-07-20T01:11:46.987 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-07-20T01:11:46.988 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-07-20T01:11:46.988 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'tag': None, 'wait_for_package': False} 2023-07-20T01:11:46.989 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T01:11:46.991 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'tag': None, 'wait_for_package': False} 2023-07-20T01:11:46.991 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T01:11:46.993 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'tag': None, 'wait_for_package': False} 2023-07-20T01:11:46.994 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T01:11:47.578 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/ 2023-07-20T01:11:47.579 INFO:teuthology.task.install.rpm:Package version is 18.0.0-5040.g1b0b2eb0 2023-07-20T01:11:47.584 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/ 2023-07-20T01:11:47.584 INFO:teuthology.task.install.rpm:Package version is 18.0.0-5040.g1b0b2eb0 2023-07-20T01:11:47.593 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/ 2023-07-20T01:11:47.594 INFO:teuthology.task.install.rpm:Package version is 18.0.0-5040.g1b0b2eb0 2023-07-20T01:11:48.103 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-20T01:11:48.103 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-07-20T01:11:48.104 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-20T01:11:48.116 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-20T01:11:48.117 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2023-07-20T01:11:48.117 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-20T01:11:48.119 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-20T01:11:48.119 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-07-20T01:11:48.120 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-20T01:11:48.135 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-20T01:11:48.136 DEBUG:teuthology.orchestra.run.smithi136:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1b0b2eb005a7c51d97a788b9900d511e1394b537/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-20T01:11:48.152 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-20T01:11:48.152 DEBUG:teuthology.orchestra.run.smithi183:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1b0b2eb005a7c51d97a788b9900d511e1394b537/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-20T01:11:48.153 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-20T01:11:48.153 DEBUG:teuthology.orchestra.run.smithi060:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1b0b2eb005a7c51d97a788b9900d511e1394b537/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-20T01:11:48.206 DEBUG:teuthology.orchestra.run.smithi136:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-20T01:11:48.223 DEBUG:teuthology.orchestra.run.smithi183:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-20T01:11:48.225 DEBUG:teuthology.orchestra.run.smithi060:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-20T01:11:48.286 DEBUG:teuthology.orchestra.run.smithi136:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-20T01:11:48.303 DEBUG:teuthology.orchestra.run.smithi183:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-20T01:11:48.307 DEBUG:teuthology.orchestra.run.smithi060:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-20T01:11:48.352 INFO:teuthology.orchestra.run.smithi136.stdout:check_obsoletes = 1 2023-07-20T01:11:48.354 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum clean all 2023-07-20T01:11:48.371 INFO:teuthology.orchestra.run.smithi183.stdout:check_obsoletes = 1 2023-07-20T01:11:48.372 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum clean all 2023-07-20T01:11:48.377 INFO:teuthology.orchestra.run.smithi060.stdout:check_obsoletes = 1 2023-07-20T01:11:48.378 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum clean all 2023-07-20T01:11:48.725 INFO:teuthology.orchestra.run.smithi136.stdout:Updating Subscription Management repositories. 2023-07-20T01:11:48.747 INFO:teuthology.orchestra.run.smithi183.stdout:Updating Subscription Management repositories. 2023-07-20T01:11:48.773 INFO:teuthology.orchestra.run.smithi060.stdout:Updating Subscription Management repositories. 2023-07-20T01:11:49.635 INFO:teuthology.orchestra.run.smithi136.stdout:44 files removed 2023-07-20T01:11:49.672 INFO:teuthology.orchestra.run.smithi183.stdout:44 files removed 2023-07-20T01:11:49.687 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-20T01:11:49.725 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-20T01:11:49.758 INFO:teuthology.orchestra.run.smithi060.stdout:44 files removed 2023-07-20T01:11:49.813 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-20T01:11:50.018 INFO:teuthology.orchestra.run.smithi136.stdout:Updating Subscription Management repositories. 2023-07-20T01:11:50.056 INFO:teuthology.orchestra.run.smithi183.stdout:Updating Subscription Management repositories. 2023-07-20T01:11:50.167 INFO:teuthology.orchestra.run.smithi060.stdout:Updating Subscription Management repositories. 2023-07-20T01:11:52.266 INFO:teuthology.orchestra.run.smithi136.stdout:ceph packages for x86_64 66 kB/s | 81 kB 00:01 2023-07-20T01:11:52.339 INFO:teuthology.orchestra.run.smithi060.stdout:ceph packages for x86_64 69 kB/s | 81 kB 00:01 2023-07-20T01:11:52.395 INFO:teuthology.orchestra.run.smithi183.stdout:ceph packages for x86_64 63 kB/s | 81 kB 00:01 2023-07-20T01:11:53.342 INFO:teuthology.orchestra.run.smithi136.stdout:ceph noarch packages 9.0 kB/s | 9.4 kB 00:01 2023-07-20T01:11:53.414 INFO:teuthology.orchestra.run.smithi060.stdout:ceph noarch packages 9.0 kB/s | 9.4 kB 00:01 2023-07-20T01:11:53.446 INFO:teuthology.orchestra.run.smithi183.stdout:ceph noarch packages 9.2 kB/s | 9.4 kB 00:01 2023-07-20T01:11:54.301 INFO:teuthology.orchestra.run.smithi136.stdout:ceph source packages 1.6 kB/s | 1.5 kB 00:00 2023-07-20T01:11:54.357 INFO:teuthology.orchestra.run.smithi060.stdout:ceph source packages 1.6 kB/s | 1.5 kB 00:00 2023-07-20T01:11:54.405 INFO:teuthology.orchestra.run.smithi183.stdout:ceph source packages 1.6 kB/s | 1.5 kB 00:00 2023-07-20T01:11:54.901 INFO:teuthology.orchestra.run.smithi136.stdout:Copr repo for python3-asyncssh owned by ceph 6.0 kB/s | 3.5 kB 00:00 2023-07-20T01:11:54.941 INFO:teuthology.orchestra.run.smithi060.stdout:Copr repo for python3-asyncssh owned by ceph 6.2 kB/s | 3.5 kB 00:00 2023-07-20T01:11:55.013 INFO:teuthology.orchestra.run.smithi183.stdout:Copr repo for python3-asyncssh owned by ceph 5.9 kB/s | 3.5 kB 00:00 2023-07-20T01:11:59.972 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 3.2 MB/s | 16 MB 00:04 2023-07-20T01:12:03.405 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 493 kB/s | 24 kB 00:00 2023-07-20T01:12:04.324 INFO:teuthology.orchestra.run.smithi060.stdout:Extra Packages for Enterprise Linux 1.7 MB/s | 16 MB 00:09 2023-07-20T01:12:04.523 INFO:teuthology.orchestra.run.smithi183.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 45 MB/s | 47 MB 00:01 2023-07-20T01:12:07.924 INFO:teuthology.orchestra.run.smithi060.stdout:lab-extras 510 kB/s | 24 kB 00:00 2023-07-20T01:12:09.276 INFO:teuthology.orchestra.run.smithi060.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 42 MB/s | 53 MB 00:01 2023-07-20T01:12:12.600 INFO:teuthology.orchestra.run.smithi183.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 50 MB/s | 53 MB 00:01 2023-07-20T01:12:17.969 INFO:teuthology.orchestra.run.smithi060.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 50 MB/s | 47 MB 00:00 2023-07-20T01:12:23.713 INFO:teuthology.orchestra.run.smithi183.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T01:12:23.714 INFO:teuthology.orchestra.run.smithi183.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T01:12:23.728 INFO:teuthology.orchestra.run.smithi183.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-20T01:12:23.729 INFO:teuthology.orchestra.run.smithi183.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-20T01:12:23.730 INFO:teuthology.orchestra.run.smithi183.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-20T01:12:23.731 INFO:teuthology.orchestra.run.smithi183.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-20T01:12:23.733 INFO:teuthology.orchestra.run.smithi183.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-20T01:12:23.737 INFO:teuthology.orchestra.run.smithi183.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-20T01:12:23.738 INFO:teuthology.orchestra.run.smithi183.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-20T01:12:23.870 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-07-20T01:12:23.882 INFO:teuthology.orchestra.run.smithi183.stdout:=========================================================================================================================== 2023-07-20T01:12:23.883 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2023-07-20T01:12:23.883 INFO:teuthology.orchestra.run.smithi183.stdout:=========================================================================================================================== 2023-07-20T01:12:23.883 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-07-20T01:12:23.883 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-20T01:12:23.883 INFO:teuthology.orchestra.run.smithi183.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-20T01:12:23.883 INFO:teuthology.orchestra.run.smithi183.stdout: ceph x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.4 k 2023-07-20T01:12:23.884 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 5.4 M 2023-07-20T01:12:23.884 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 850 k 2023-07-20T01:12:23.884 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 167 k 2023-07-20T01:12:23.884 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 1.6 M 2023-07-20T01:12:23.884 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 129 k 2023-07-20T01:12:23.884 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 1.7 M 2023-07-20T01:12:23.884 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 7.4 M 2023-07-20T01:12:23.885 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-rook noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 47 k 2023-07-20T01:12:23.885 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 14 M 2023-07-20T01:12:23.885 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 48 M 2023-07-20T01:12:23.885 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-volume noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 258 k 2023-07-20T01:12:23.885 INFO:teuthology.orchestra.run.smithi183.stdout: cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 165 k 2023-07-20T01:12:23.885 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 260 k 2023-07-20T01:12:23.885 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-top noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 18 k 2023-07-20T01:12:23.885 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-20T01:12:23.886 INFO:teuthology.orchestra.run.smithi183.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-20T01:12:23.886 INFO:teuthology.orchestra.run.smithi183.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-20T01:12:23.886 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 31 k 2023-07-20T01:12:23.886 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 771 k 2023-07-20T01:12:23.886 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T01:12:23.886 INFO:teuthology.orchestra.run.smithi183.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-20T01:12:23.886 INFO:teuthology.orchestra.run.smithi183.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-20T01:12:23.887 INFO:teuthology.orchestra.run.smithi183.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-20T01:12:23.887 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-20T01:12:23.887 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-20T01:12:23.887 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-20T01:12:23.887 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 197 k 2023-07-20T01:12:23.887 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 370 k 2023-07-20T01:12:23.887 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 363 k 2023-07-20T01:12:23.887 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 107 k 2023-07-20T01:12:23.888 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 94 k 2023-07-20T01:12:23.888 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.5 M 2023-07-20T01:12:23.888 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 179 k 2023-07-20T01:12:23.888 INFO:teuthology.orchestra.run.smithi183.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-20T01:12:23.888 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrading: 2023-07-20T01:12:23.888 INFO:teuthology.orchestra.run.smithi183.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-20T01:12:23.888 INFO:teuthology.orchestra.run.smithi183.stdout: librados2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.7 M 2023-07-20T01:12:23.889 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.6 M 2023-07-20T01:12:23.889 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2023-07-20T01:12:23.889 INFO:teuthology.orchestra.run.smithi183.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-20T01:12:23.889 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 22 M 2023-07-20T01:12:23.889 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-grafana-dashboards noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 23 k 2023-07-20T01:12:23.889 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 2.3 M 2023-07-20T01:12:23.889 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 241 k 2023-07-20T01:12:23.889 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 4.6 M 2023-07-20T01:12:23.890 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 18 M 2023-07-20T01:12:23.890 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-prometheus-alerts noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 15 k 2023-07-20T01:12:23.890 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-selinux x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 24 k 2023-07-20T01:12:23.890 INFO:teuthology.orchestra.run.smithi183.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-20T01:12:23.890 INFO:teuthology.orchestra.run.smithi183.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-20T01:12:23.890 INFO:teuthology.orchestra.run.smithi183.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-20T01:12:23.890 INFO:teuthology.orchestra.run.smithi183.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-20T01:12:23.890 INFO:teuthology.orchestra.run.smithi183.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-20T01:12:23.891 INFO:teuthology.orchestra.run.smithi183.stdout: libcephsqlite x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T01:12:23.891 INFO:teuthology.orchestra.run.smithi183.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-20T01:12:23.891 INFO:teuthology.orchestra.run.smithi183.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-20T01:12:23.891 INFO:teuthology.orchestra.run.smithi183.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-20T01:12:23.891 INFO:teuthology.orchestra.run.smithi183.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-20T01:12:23.891 INFO:teuthology.orchestra.run.smithi183.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-20T01:12:23.891 INFO:teuthology.orchestra.run.smithi183.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-20T01:12:23.892 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 472 k 2023-07-20T01:12:23.892 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-20T01:12:23.892 INFO:teuthology.orchestra.run.smithi183.stdout: librgw2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.8 M 2023-07-20T01:12:23.892 INFO:teuthology.orchestra.run.smithi183.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-20T01:12:23.892 INFO:teuthology.orchestra.run.smithi183.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-20T01:12:23.892 INFO:teuthology.orchestra.run.smithi183.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-20T01:12:23.892 INFO:teuthology.orchestra.run.smithi183.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-20T01:12:23.893 INFO:teuthology.orchestra.run.smithi183.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-20T01:12:23.893 INFO:teuthology.orchestra.run.smithi183.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-20T01:12:23.893 INFO:teuthology.orchestra.run.smithi183.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-20T01:12:23.893 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-20T01:12:23.893 INFO:teuthology.orchestra.run.smithi183.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-20T01:12:23.893 INFO:teuthology.orchestra.run.smithi183.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-20T01:12:23.893 INFO:teuthology.orchestra.run.smithi183.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-20T01:12:23.893 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-20T01:12:23.894 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-20T01:12:23.894 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-20T01:12:23.894 INFO:teuthology.orchestra.run.smithi183.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-20T01:12:23.894 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-20T01:12:23.894 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 45 k 2023-07-20T01:12:23.894 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 122 k 2023-07-20T01:12:23.894 INFO:teuthology.orchestra.run.smithi183.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-20T01:12:23.895 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-20T01:12:23.895 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-20T01:12:23.895 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-20T01:12:23.895 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-20T01:12:23.895 INFO:teuthology.orchestra.run.smithi183.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-20T01:12:23.895 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-20T01:12:23.895 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-20T01:12:23.895 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-20T01:12:23.896 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-20T01:12:23.896 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-20T01:12:23.896 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T01:12:23.896 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-20T01:12:23.896 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-20T01:12:23.896 INFO:teuthology.orchestra.run.smithi183.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-20T01:12:23.896 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-20T01:12:23.897 INFO:teuthology.orchestra.run.smithi183.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-20T01:12:23.897 INFO:teuthology.orchestra.run.smithi183.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-20T01:12:23.897 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-20T01:12:23.897 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-20T01:12:23.897 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-20T01:12:23.897 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-20T01:12:23.897 INFO:teuthology.orchestra.run.smithi183.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-20T01:12:23.897 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-20T01:12:23.898 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-20T01:12:23.898 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-20T01:12:23.898 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-20T01:12:23.898 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-20T01:12:23.898 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-20T01:12:23.898 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-20T01:12:23.898 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T01:12:23.899 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-20T01:12:23.899 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-20T01:12:23.899 INFO:teuthology.orchestra.run.smithi183.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-20T01:12:23.899 INFO:teuthology.orchestra.run.smithi183.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-20T01:12:23.899 INFO:teuthology.orchestra.run.smithi183.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-20T01:12:23.899 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-20T01:12:23.899 INFO:teuthology.orchestra.run.smithi183.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-20T01:12:23.899 INFO:teuthology.orchestra.run.smithi183.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-20T01:12:23.900 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-20T01:12:23.900 INFO:teuthology.orchestra.run.smithi183.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-20T01:12:23.900 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-20T01:12:23.900 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-20T01:12:23.900 INFO:teuthology.orchestra.run.smithi183.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-20T01:12:23.900 INFO:teuthology.orchestra.run.smithi183.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-20T01:12:23.900 INFO:teuthology.orchestra.run.smithi183.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-20T01:12:23.901 INFO:teuthology.orchestra.run.smithi183.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-20T01:12:23.901 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-20T01:12:23.901 INFO:teuthology.orchestra.run.smithi183.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-20T01:12:23.901 INFO:teuthology.orchestra.run.smithi183.stdout:Enabling module streams: 2023-07-20T01:12:23.901 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql 10 2023-07-20T01:12:23.901 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-07-20T01:12:23.901 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-07-20T01:12:23.901 INFO:teuthology.orchestra.run.smithi183.stdout:=========================================================================================================================== 2023-07-20T01:12:23.902 INFO:teuthology.orchestra.run.smithi183.stdout:Install 128 Packages 2023-07-20T01:12:23.902 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrade 3 Packages 2023-07-20T01:12:23.902 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-07-20T01:12:23.902 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 206 M 2023-07-20T01:12:23.903 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-07-20T01:12:24.832 INFO:teuthology.orchestra.run.smithi183.stdout:(1/131): ceph-18.0.0-5040.g1b0b2eb0.el8.x86_64. 16 kB/s | 6.4 kB 00:00 2023-07-20T01:12:25.766 INFO:teuthology.orchestra.run.smithi183.stdout:(2/131): ceph-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 911 kB/s | 850 kB 00:00 2023-07-20T01:12:25.908 INFO:teuthology.orchestra.run.smithi183.stdout:(3/131): ceph-immutable-object-cache-18.0.0-504 1.2 MB/s | 167 kB 00:00 2023-07-20T01:12:27.116 INFO:teuthology.orchestra.run.smithi183.stdout:(4/131): ceph-mds-18.0.0-5040.g1b0b2eb0.el8.x86 1.9 MB/s | 2.3 MB 00:01 2023-07-20T01:12:27.567 INFO:teuthology.orchestra.run.smithi183.stdout:(5/131): ceph-mgr-18.0.0-5040.g1b0b2eb0.el8.x86 3.5 MB/s | 1.6 MB 00:00 2023-07-20T01:12:28.142 INFO:teuthology.orchestra.run.smithi183.stdout:(6/131): ceph-base-18.0.0-5040.g1b0b2eb0.el8.x8 1.5 MB/s | 5.4 MB 00:03 2023-07-20T01:12:28.517 INFO:teuthology.orchestra.run.smithi183.stdout:(7/131): ceph-mon-18.0.0-5040.g1b0b2eb0.el8.x86 4.8 MB/s | 4.6 MB 00:00 2023-07-20T01:12:29.535 INFO:teuthology.orchestra.run.smithi060.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T01:12:29.536 INFO:teuthology.orchestra.run.smithi060.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T01:12:29.551 INFO:teuthology.orchestra.run.smithi060.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-20T01:12:29.552 INFO:teuthology.orchestra.run.smithi060.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-20T01:12:29.553 INFO:teuthology.orchestra.run.smithi060.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-20T01:12:29.555 INFO:teuthology.orchestra.run.smithi060.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-20T01:12:29.557 INFO:teuthology.orchestra.run.smithi060.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-20T01:12:29.560 INFO:teuthology.orchestra.run.smithi060.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-20T01:12:29.561 INFO:teuthology.orchestra.run.smithi060.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-20T01:12:29.705 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2023-07-20T01:12:29.719 INFO:teuthology.orchestra.run.smithi060.stdout:=========================================================================================================================== 2023-07-20T01:12:29.719 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2023-07-20T01:12:29.719 INFO:teuthology.orchestra.run.smithi060.stdout:=========================================================================================================================== 2023-07-20T01:12:29.719 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2023-07-20T01:12:29.719 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-20T01:12:29.720 INFO:teuthology.orchestra.run.smithi060.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-20T01:12:29.720 INFO:teuthology.orchestra.run.smithi060.stdout: ceph x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.4 k 2023-07-20T01:12:29.720 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 5.4 M 2023-07-20T01:12:29.720 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 850 k 2023-07-20T01:12:29.720 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 167 k 2023-07-20T01:12:29.720 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 1.6 M 2023-07-20T01:12:29.721 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 129 k 2023-07-20T01:12:29.721 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-dashboard noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 1.7 M 2023-07-20T01:12:29.721 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 7.4 M 2023-07-20T01:12:29.721 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-rook noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 47 k 2023-07-20T01:12:29.721 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-radosgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 14 M 2023-07-20T01:12:29.721 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-test x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 48 M 2023-07-20T01:12:29.722 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-volume noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 258 k 2023-07-20T01:12:29.722 INFO:teuthology.orchestra.run.smithi060.stdout: cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 165 k 2023-07-20T01:12:29.722 INFO:teuthology.orchestra.run.smithi060.stdout: cephfs-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 260 k 2023-07-20T01:12:29.722 INFO:teuthology.orchestra.run.smithi060.stdout: cephfs-top noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 18 k 2023-07-20T01:12:29.722 INFO:teuthology.orchestra.run.smithi060.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-20T01:12:29.722 INFO:teuthology.orchestra.run.smithi060.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-20T01:12:29.722 INFO:teuthology.orchestra.run.smithi060.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-20T01:12:29.723 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 31 k 2023-07-20T01:12:29.723 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 771 k 2023-07-20T01:12:29.723 INFO:teuthology.orchestra.run.smithi060.stdout: librados-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T01:12:29.723 INFO:teuthology.orchestra.run.smithi060.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-20T01:12:29.723 INFO:teuthology.orchestra.run.smithi060.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-20T01:12:29.723 INFO:teuthology.orchestra.run.smithi060.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-20T01:12:29.724 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-20T01:12:29.724 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-20T01:12:29.724 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-20T01:12:29.724 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cephfs x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 197 k 2023-07-20T01:12:29.724 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rados x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 370 k 2023-07-20T01:12:29.724 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 363 k 2023-07-20T01:12:29.725 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 107 k 2023-07-20T01:12:29.725 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 94 k 2023-07-20T01:12:29.725 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.5 M 2023-07-20T01:12:29.725 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-nbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 179 k 2023-07-20T01:12:29.725 INFO:teuthology.orchestra.run.smithi060.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-20T01:12:29.725 INFO:teuthology.orchestra.run.smithi060.stdout:Upgrading: 2023-07-20T01:12:29.726 INFO:teuthology.orchestra.run.smithi060.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-20T01:12:29.726 INFO:teuthology.orchestra.run.smithi060.stdout: librados2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.7 M 2023-07-20T01:12:29.726 INFO:teuthology.orchestra.run.smithi060.stdout: librbd1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.6 M 2023-07-20T01:12:29.726 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2023-07-20T01:12:29.726 INFO:teuthology.orchestra.run.smithi060.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-20T01:12:29.726 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 22 M 2023-07-20T01:12:29.726 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-grafana-dashboards noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 23 k 2023-07-20T01:12:29.727 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mds x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 2.3 M 2023-07-20T01:12:29.727 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-modules-core noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 241 k 2023-07-20T01:12:29.727 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mon x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 4.6 M 2023-07-20T01:12:29.727 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-osd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 18 M 2023-07-20T01:12:29.727 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-prometheus-alerts noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 15 k 2023-07-20T01:12:29.728 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-selinux x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 24 k 2023-07-20T01:12:29.728 INFO:teuthology.orchestra.run.smithi060.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-20T01:12:29.728 INFO:teuthology.orchestra.run.smithi060.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-20T01:12:29.728 INFO:teuthology.orchestra.run.smithi060.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-20T01:12:29.728 INFO:teuthology.orchestra.run.smithi060.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-20T01:12:29.728 INFO:teuthology.orchestra.run.smithi060.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-20T01:12:29.729 INFO:teuthology.orchestra.run.smithi060.stdout: libcephsqlite x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T01:12:29.729 INFO:teuthology.orchestra.run.smithi060.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-20T01:12:29.729 INFO:teuthology.orchestra.run.smithi060.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-20T01:12:29.729 INFO:teuthology.orchestra.run.smithi060.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-20T01:12:29.729 INFO:teuthology.orchestra.run.smithi060.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-20T01:12:29.729 INFO:teuthology.orchestra.run.smithi060.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-20T01:12:29.730 INFO:teuthology.orchestra.run.smithi060.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-20T01:12:29.730 INFO:teuthology.orchestra.run.smithi060.stdout: libradosstriper1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 472 k 2023-07-20T01:12:29.730 INFO:teuthology.orchestra.run.smithi060.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-20T01:12:29.730 INFO:teuthology.orchestra.run.smithi060.stdout: librgw2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.8 M 2023-07-20T01:12:29.730 INFO:teuthology.orchestra.run.smithi060.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-20T01:12:29.730 INFO:teuthology.orchestra.run.smithi060.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-20T01:12:29.730 INFO:teuthology.orchestra.run.smithi060.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-20T01:12:29.731 INFO:teuthology.orchestra.run.smithi060.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-20T01:12:29.731 INFO:teuthology.orchestra.run.smithi060.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-20T01:12:29.731 INFO:teuthology.orchestra.run.smithi060.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-20T01:12:29.731 INFO:teuthology.orchestra.run.smithi060.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-20T01:12:29.731 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-20T01:12:29.731 INFO:teuthology.orchestra.run.smithi060.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-20T01:12:29.731 INFO:teuthology.orchestra.run.smithi060.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-20T01:12:29.732 INFO:teuthology.orchestra.run.smithi060.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-20T01:12:29.732 INFO:teuthology.orchestra.run.smithi060.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-20T01:12:29.732 INFO:teuthology.orchestra.run.smithi060.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-20T01:12:29.732 INFO:teuthology.orchestra.run.smithi060.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-20T01:12:29.733 INFO:teuthology.orchestra.run.smithi060.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-20T01:12:29.733 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-20T01:12:29.733 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-argparse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 45 k 2023-07-20T01:12:29.733 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 122 k 2023-07-20T01:12:29.733 INFO:teuthology.orchestra.run.smithi060.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-20T01:12:29.734 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-20T01:12:29.734 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-20T01:12:29.734 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-20T01:12:29.734 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-20T01:12:29.734 INFO:teuthology.orchestra.run.smithi060.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-20T01:12:29.734 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-20T01:12:29.734 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-20T01:12:29.735 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-20T01:12:29.735 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-20T01:12:29.735 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-20T01:12:29.735 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T01:12:29.735 INFO:teuthology.orchestra.run.smithi060.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-20T01:12:29.735 INFO:teuthology.orchestra.run.smithi060.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-20T01:12:29.735 INFO:teuthology.orchestra.run.smithi060.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-20T01:12:29.736 INFO:teuthology.orchestra.run.smithi060.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-20T01:12:29.736 INFO:teuthology.orchestra.run.smithi060.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-20T01:12:29.736 INFO:teuthology.orchestra.run.smithi060.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-20T01:12:29.736 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-20T01:12:29.736 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-20T01:12:29.736 INFO:teuthology.orchestra.run.smithi060.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-20T01:12:29.736 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-20T01:12:29.737 INFO:teuthology.orchestra.run.smithi060.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-20T01:12:29.737 INFO:teuthology.orchestra.run.smithi060.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-20T01:12:29.737 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-20T01:12:29.737 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-20T01:12:29.737 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-20T01:12:29.737 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-20T01:12:29.738 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-20T01:12:29.738 INFO:teuthology.orchestra.run.smithi060.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-20T01:12:29.738 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T01:12:29.738 INFO:teuthology.orchestra.run.smithi060.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-20T01:12:29.738 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-20T01:12:29.738 INFO:teuthology.orchestra.run.smithi060.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-20T01:12:29.738 INFO:teuthology.orchestra.run.smithi060.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-20T01:12:29.739 INFO:teuthology.orchestra.run.smithi060.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-20T01:12:29.739 INFO:teuthology.orchestra.run.smithi060.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-20T01:12:29.739 INFO:teuthology.orchestra.run.smithi060.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-20T01:12:29.739 INFO:teuthology.orchestra.run.smithi060.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-20T01:12:29.739 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-20T01:12:29.739 INFO:teuthology.orchestra.run.smithi060.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-20T01:12:29.739 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-20T01:12:29.740 INFO:teuthology.orchestra.run.smithi060.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-20T01:12:29.740 INFO:teuthology.orchestra.run.smithi060.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-20T01:12:29.740 INFO:teuthology.orchestra.run.smithi060.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-20T01:12:29.740 INFO:teuthology.orchestra.run.smithi060.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-20T01:12:29.740 INFO:teuthology.orchestra.run.smithi060.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-20T01:12:29.740 INFO:teuthology.orchestra.run.smithi060.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-20T01:12:29.740 INFO:teuthology.orchestra.run.smithi060.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-20T01:12:29.741 INFO:teuthology.orchestra.run.smithi060.stdout:Enabling module streams: 2023-07-20T01:12:29.741 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql 10 2023-07-20T01:12:29.741 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:12:29.741 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2023-07-20T01:12:29.741 INFO:teuthology.orchestra.run.smithi060.stdout:=========================================================================================================================== 2023-07-20T01:12:29.741 INFO:teuthology.orchestra.run.smithi060.stdout:Install 128 Packages 2023-07-20T01:12:29.741 INFO:teuthology.orchestra.run.smithi060.stdout:Upgrade 3 Packages 2023-07-20T01:12:29.742 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:12:29.742 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 206 M 2023-07-20T01:12:29.742 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2023-07-20T01:12:30.284 INFO:teuthology.orchestra.run.smithi183.stdout:(8/131): ceph-radosgw-18.0.0-5040.g1b0b2eb0.el8 7.7 MB/s | 14 MB 00:01 2023-07-20T01:12:30.417 INFO:teuthology.orchestra.run.smithi183.stdout:(9/131): ceph-selinux-18.0.0-5040.g1b0b2eb0.el8 182 kB/s | 24 kB 00:00 2023-07-20T01:12:30.969 INFO:teuthology.orchestra.run.smithi060.stdout:(1/131): ceph-18.0.0-5040.g1b0b2eb0.el8.x86_64. 12 kB/s | 6.4 kB 00:00 2023-07-20T01:12:31.734 INFO:teuthology.orchestra.run.smithi183.stdout:(10/131): ceph-osd-18.0.0-5040.g1b0b2eb0.el8.x8 4.9 MB/s | 18 MB 00:03 2023-07-20T01:12:31.893 INFO:teuthology.orchestra.run.smithi183.stdout:(11/131): cephfs-mirror-18.0.0-5040.g1b0b2eb0.e 1.6 MB/s | 260 kB 00:00 2023-07-20T01:12:32.101 INFO:teuthology.orchestra.run.smithi183.stdout:(12/131): ceph-common-18.0.0-5040.g1b0b2eb0.el8 2.9 MB/s | 22 MB 00:07 2023-07-20T01:12:32.118 INFO:teuthology.orchestra.run.smithi183.stdout:(13/131): libcephfs-devel-18.0.0-5040.g1b0b2eb0 138 kB/s | 31 kB 00:00 2023-07-20T01:12:32.260 INFO:teuthology.orchestra.run.smithi183.stdout:(14/131): libcephsqlite-18.0.0-5040.g1b0b2eb0.e 980 kB/s | 139 kB 00:00 2023-07-20T01:12:32.477 INFO:teuthology.orchestra.run.smithi183.stdout:(15/131): libcephfs2-18.0.0-5040.g1b0b2eb0.el8. 2.0 MB/s | 771 kB 00:00 2023-07-20T01:12:32.702 INFO:teuthology.orchestra.run.smithi183.stdout:(16/131): librados-devel-18.0.0-5040.g1b0b2eb0. 314 kB/s | 139 kB 00:00 2023-07-20T01:12:32.744 INFO:teuthology.orchestra.run.smithi183.stdout:(17/131): libradosstriper1-18.0.0-5040.g1b0b2eb 1.7 MB/s | 472 kB 00:00 2023-07-20T01:12:32.836 INFO:teuthology.orchestra.run.smithi183.stdout:(18/131): python3-ceph-argparse-18.0.0-5040.g1b 487 kB/s | 45 kB 00:00 2023-07-20T01:12:32.945 INFO:teuthology.orchestra.run.smithi183.stdout:(19/131): python3-ceph-common-18.0.0-5040.g1b0b 1.1 MB/s | 122 kB 00:00 2023-07-20T01:12:33.211 INFO:teuthology.orchestra.run.smithi183.stdout:(20/131): python3-cephfs-18.0.0-5040.g1b0b2eb0. 738 kB/s | 197 kB 00:00 2023-07-20T01:12:33.495 INFO:teuthology.orchestra.run.smithi183.stdout:(21/131): python3-rados-18.0.0-5040.g1b0b2eb0.e 1.3 MB/s | 370 kB 00:00 2023-07-20T01:12:33.686 INFO:teuthology.orchestra.run.smithi060.stdout:(2/131): ceph-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 313 kB/s | 850 kB 00:02 2023-07-20T01:12:33.720 INFO:teuthology.orchestra.run.smithi183.stdout:(22/131): python3-rbd-18.0.0-5040.g1b0b2eb0.el8 1.6 MB/s | 363 kB 00:00 2023-07-20T01:12:33.796 INFO:teuthology.orchestra.run.smithi183.stdout:(23/131): python3-rgw-18.0.0-5040.g1b0b2eb0.el8 1.4 MB/s | 107 kB 00:00 2023-07-20T01:12:33.896 INFO:teuthology.orchestra.run.smithi183.stdout:(24/131): rbd-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 937 kB/s | 94 kB 00:00 2023-07-20T01:12:34.061 INFO:teuthology.orchestra.run.smithi060.stdout:(3/131): ceph-immutable-object-cache-18.0.0-504 446 kB/s | 167 kB 00:00 2023-07-20T01:12:34.787 INFO:teuthology.orchestra.run.smithi060.stdout:(4/131): ceph-base-18.0.0-5040.g1b0b2eb0.el8.x8 1.2 MB/s | 5.4 MB 00:04 2023-07-20T01:12:35.022 INFO:teuthology.orchestra.run.smithi183.stdout:(25/131): rbd-mirror-18.0.0-5040.g1b0b2eb0.el8. 3.1 MB/s | 3.5 MB 00:01 2023-07-20T01:12:35.096 INFO:teuthology.orchestra.run.smithi183.stdout:(26/131): rbd-nbd-18.0.0-5040.g1b0b2eb0.el8.x86 2.3 MB/s | 179 kB 00:00 2023-07-20T01:12:35.163 INFO:teuthology.orchestra.run.smithi183.stdout:(27/131): ceph-grafana-dashboards-18.0.0-5040.g 346 kB/s | 23 kB 00:00 2023-07-20T01:12:35.247 INFO:teuthology.orchestra.run.smithi183.stdout:(28/131): ceph-mgr-cephadm-18.0.0-5040.g1b0b2eb 1.5 MB/s | 129 kB 00:00 2023-07-20T01:12:35.595 INFO:teuthology.orchestra.run.smithi060.stdout:(5/131): ceph-mgr-18.0.0-5040.g1b0b2eb0.el8.x86 1.9 MB/s | 1.6 MB 00:00 2023-07-20T01:12:35.706 INFO:teuthology.orchestra.run.smithi183.stdout:(29/131): ceph-mgr-dashboard-18.0.0-5040.g1b0b2 3.7 MB/s | 1.7 MB 00:00 2023-07-20T01:12:35.781 INFO:teuthology.orchestra.run.smithi183.stdout:(30/131): librgw2-18.0.0-5040.g1b0b2eb0.el8.x86 2.2 MB/s | 6.8 MB 00:03 2023-07-20T01:12:35.948 INFO:teuthology.orchestra.run.smithi183.stdout:(31/131): ceph-mgr-modules-core-18.0.0-5040.g1b 1.4 MB/s | 241 kB 00:00 2023-07-20T01:12:36.015 INFO:teuthology.orchestra.run.smithi183.stdout:(32/131): ceph-mgr-rook-18.0.0-5040.g1b0b2eb0.e 711 kB/s | 47 kB 00:00 2023-07-20T01:12:36.098 INFO:teuthology.orchestra.run.smithi183.stdout:(33/131): ceph-prometheus-alerts-18.0.0-5040.g1 175 kB/s | 15 kB 00:00 2023-07-20T01:12:36.282 INFO:teuthology.orchestra.run.smithi183.stdout:(34/131): ceph-volume-18.0.0-5040.g1b0b2eb0.el8 1.4 MB/s | 258 kB 00:00 2023-07-20T01:12:36.407 INFO:teuthology.orchestra.run.smithi183.stdout:(35/131): cephadm-18.0.0-5040.g1b0b2eb0.el8.noa 1.3 MB/s | 165 kB 00:00 2023-07-20T01:12:36.412 INFO:teuthology.orchestra.run.smithi060.stdout:(6/131): ceph-mds-18.0.0-5040.g1b0b2eb0.el8.x86 997 kB/s | 2.3 MB 00:02 2023-07-20T01:12:36.558 INFO:teuthology.orchestra.run.smithi183.stdout:(36/131): cephfs-top-18.0.0-5040.g1b0b2eb0.el8. 120 kB/s | 18 kB 00:00 2023-07-20T01:12:36.966 INFO:teuthology.orchestra.run.smithi183.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 750 kB/s | 306 kB 00:00 2023-07-20T01:12:37.058 INFO:teuthology.orchestra.run.smithi183.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 643 kB/s | 59 kB 00:00 2023-07-20T01:12:37.167 INFO:teuthology.orchestra.run.smithi183.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 693 kB/s | 75 kB 00:00 2023-07-20T01:12:37.525 INFO:teuthology.orchestra.run.smithi183.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 1.3 MB/s | 476 kB 00:00 2023-07-20T01:12:37.592 INFO:teuthology.orchestra.run.smithi183.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 657 kB/s | 44 kB 00:00 2023-07-20T01:12:37.726 INFO:teuthology.orchestra.run.smithi183.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.4 MB/s | 185 kB 00:00 2023-07-20T01:12:37.810 INFO:teuthology.orchestra.run.smithi183.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 396 kB/s | 33 kB 00:00 2023-07-20T01:12:37.901 INFO:teuthology.orchestra.run.smithi183.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 178 kB/s | 16 kB 00:00 2023-07-20T01:12:38.027 INFO:teuthology.orchestra.run.smithi183.stdout:(45/131): ceph-mgr-diskprediction-local-18.0.0- 3.2 MB/s | 7.4 MB 00:02 2023-07-20T01:12:38.085 INFO:teuthology.orchestra.run.smithi183.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 939 kB/s | 173 kB 00:00 2023-07-20T01:12:38.179 INFO:teuthology.orchestra.run.smithi060.stdout:(7/131): ceph-mon-18.0.0-5040.g1b0b2eb0.el8.x86 1.8 MB/s | 4.6 MB 00:02 2023-07-20T01:12:38.194 INFO:teuthology.orchestra.run.smithi183.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 751 kB/s | 81 kB 00:00 2023-07-20T01:12:38.303 INFO:teuthology.orchestra.run.smithi183.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 102 kB/s | 11 kB 00:00 2023-07-20T01:12:38.412 INFO:teuthology.orchestra.run.smithi183.stdout:(49/131): python3-jaraco-functools-2.0-4.el8.no 162 kB/s | 18 kB 00:00 2023-07-20T01:12:38.612 INFO:teuthology.orchestra.run.smithi183.stdout:(50/131): python3-cherrypy-18.4.0-1.el8.noarch. 658 kB/s | 384 kB 00:00 2023-07-20T01:12:38.720 INFO:teuthology.orchestra.run.smithi183.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 451 kB/s | 49 kB 00:00 2023-07-20T01:12:38.812 INFO:teuthology.orchestra.run.smithi183.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 640 kB/s | 59 kB 00:00 2023-07-20T01:12:38.912 INFO:teuthology.orchestra.run.smithi183.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 604 kB/s | 60 kB 00:00 2023-07-20T01:12:39.079 INFO:teuthology.orchestra.run.smithi183.stdout:(54/131): python3-pecan-1.3.2-9.el8.noarch.rpm 1.7 MB/s | 283 kB 00:00 2023-07-20T01:12:39.130 INFO:teuthology.orchestra.run.smithi183.stdout:(55/131): python3-kubernetes-11.0.0-6.el8.noarc 1.6 MB/s | 1.2 MB 00:00 2023-07-20T01:12:39.172 INFO:teuthology.orchestra.run.smithi183.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 169 kB/s | 16 kB 00:00 2023-07-20T01:12:39.214 INFO:teuthology.orchestra.run.smithi183.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 402 kB/s | 34 kB 00:00 2023-07-20T01:12:39.305 INFO:teuthology.orchestra.run.smithi183.stdout:(58/131): python3-rsa-4.8-1.el8.noarch.rpm 671 kB/s | 61 kB 00:00 2023-07-20T01:12:39.331 INFO:teuthology.orchestra.run.smithi183.stdout:(59/131): python3-routes-2.4.1-12.el8.noarch.rp 1.2 MB/s | 196 kB 00:00 2023-07-20T01:12:39.398 INFO:teuthology.orchestra.run.smithi183.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 204 kB/s | 19 kB 00:00 2023-07-20T01:12:39.423 INFO:teuthology.orchestra.run.smithi183.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 260 kB/s | 24 kB 00:00 2023-07-20T01:12:39.482 INFO:teuthology.orchestra.run.smithi183.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 350 kB/s | 29 kB 00:00 2023-07-20T01:12:39.515 INFO:teuthology.orchestra.run.smithi183.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 299 kB/s | 27 kB 00:00 2023-07-20T01:12:39.632 INFO:teuthology.orchestra.run.smithi183.stdout:(64/131): python3-waitress-1.4.3-1.el8.noarch.r 1.7 MB/s | 255 kB 00:00 2023-07-20T01:12:39.657 INFO:teuthology.orchestra.run.smithi183.stdout:(65/131): python3-webob-1.8.5-1.el8.1.noarch.rp 1.7 MB/s | 251 kB 00:00 2023-07-20T01:12:39.716 INFO:teuthology.orchestra.run.smithi183.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 733 kB/s | 61 kB 00:00 2023-07-20T01:12:39.741 INFO:teuthology.orchestra.run.smithi183.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 1.0 MB/s | 85 kB 00:00 2023-07-20T01:12:39.808 INFO:teuthology.orchestra.run.smithi183.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 249 kB/s | 23 kB 00:00 2023-07-20T01:12:39.834 INFO:teuthology.orchestra.run.smithi183.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-07-20T01:12:40.126 INFO:teuthology.orchestra.run.smithi183.stdout:(70/131): python3-scipy-1.0.0-21.module+el8.5.0 47 MB/s | 14 MB 00:00 2023-07-20T01:12:40.168 INFO:teuthology.orchestra.run.smithi183.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 2.6 MB/s | 110 kB 00:00 2023-07-20T01:12:40.193 INFO:teuthology.orchestra.run.smithi183.stdout:(72/131): python3-pytz-2017.2-9.el8.noarch.rpm 2.1 MB/s | 54 kB 00:00 2023-07-20T01:12:40.235 INFO:teuthology.orchestra.run.smithi183.stdout:(73/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.1 MB/s | 259 kB 00:00 2023-07-20T01:12:40.277 INFO:teuthology.orchestra.run.smithi183.stdout:(74/131): socat-1.7.4.1-1.el8.x86_64.rpm 7.6 MB/s | 323 kB 00:00 2023-07-20T01:12:40.302 INFO:teuthology.orchestra.run.smithi183.stdout:(75/131): python3-markupsafe-0.23-19.el8.x86_64 1.5 MB/s | 39 kB 00:00 2023-07-20T01:12:40.344 INFO:teuthology.orchestra.run.smithi183.stdout:(76/131): bison-3.0.4-10.el8.x86_64.rpm 16 MB/s | 688 kB 00:00 2023-07-20T01:12:40.386 INFO:teuthology.orchestra.run.smithi183.stdout:(77/131): python3-jinja2-2.10.1-3.el8.noarch.rp 13 MB/s | 538 kB 00:00 2023-07-20T01:12:40.428 INFO:teuthology.orchestra.run.smithi183.stdout:(78/131): python3-mako-1.0.6-13.el8.noarch.rpm 3.7 MB/s | 157 kB 00:00 2023-07-20T01:12:40.462 INFO:teuthology.orchestra.run.smithi183.stdout:(79/131): oniguruma-6.8.2-2.el8.x86_64.rpm 5.5 MB/s | 187 kB 00:00 2023-07-20T01:12:40.512 INFO:teuthology.orchestra.run.smithi183.stdout:(80/131): thrift-0.13.0-2.el8.x86_64.rpm 2.3 MB/s | 1.7 MB 00:00 2023-07-20T01:12:40.554 INFO:teuthology.orchestra.run.smithi183.stdout:(81/131): automake-1.16.1-7.el8.noarch.rpm 7.6 MB/s | 713 kB 00:00 2023-07-20T01:12:40.604 INFO:teuthology.orchestra.run.smithi183.stdout:(82/131): python3-jsonpatch-1.21-2.el8.noarch.r 598 kB/s | 27 kB 00:00 2023-07-20T01:12:40.675 INFO:teuthology.orchestra.run.smithi183.stdout:(83/131): flex-2.6.1-9.el8.x86_64.rpm 4.4 MB/s | 320 kB 00:00 2023-07-20T01:12:40.722 INFO:teuthology.orchestra.run.smithi183.stdout:(84/131): python3-prettytable-0.7.2-14.el8.noar 954 kB/s | 44 kB 00:00 2023-07-20T01:12:40.797 INFO:teuthology.orchestra.run.smithi183.stdout:(85/131): python3-babel-2.5.1-7.el8.noarch.rpm 17 MB/s | 4.8 MB 00:00 2023-07-20T01:12:40.839 INFO:teuthology.orchestra.run.smithi183.stdout:(86/131): python3-werkzeug-0.12.2-4.el8.noarch. 11 MB/s | 457 kB 00:00 2023-07-20T01:12:40.881 INFO:teuthology.orchestra.run.smithi183.stdout:(87/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 5.3 MB/s | 225 kB 00:00 2023-07-20T01:12:40.939 INFO:teuthology.orchestra.run.smithi183.stdout:(88/131): python3-numpy-1.14.3-10.el8.x86_64.rp 17 MB/s | 3.7 MB 00:00 2023-07-20T01:12:40.965 INFO:teuthology.orchestra.run.smithi183.stdout:(89/131): uuid-1.6.2-43.el8.x86_64.rpm 765 kB/s | 64 kB 00:00 2023-07-20T01:12:40.981 INFO:teuthology.orchestra.run.smithi183.stdout:(90/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 3.0 MB/s | 126 kB 00:00 2023-07-20T01:12:41.007 INFO:teuthology.orchestra.run.smithi183.stdout:(91/131): python3-jsonpointer-1.10-11.el8.noarc 476 kB/s | 20 kB 00:00 2023-07-20T01:12:41.032 INFO:teuthology.orchestra.run.smithi183.stdout:(92/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.0 MB/s | 103 kB 00:00 2023-07-20T01:12:41.066 INFO:teuthology.orchestra.run.smithi183.stdout:(93/131): libtool-2.4.6-25.el8.x86_64.rpm 12 MB/s | 709 kB 00:00 2023-07-20T01:12:41.108 INFO:teuthology.orchestra.run.smithi183.stdout:(94/131): libpq-13.5-1.el8.x86_64.rpm 4.6 MB/s | 198 kB 00:00 2023-07-20T01:12:41.208 INFO:teuthology.orchestra.run.smithi183.stdout:(95/131): openblas-threads-0.3.15-3.el8.x86_64. 28 MB/s | 4.9 MB 00:00 2023-07-20T01:12:41.242 INFO:teuthology.orchestra.run.smithi183.stdout:(96/131): jq-1.6-3.el8.x86_64.rpm 5.9 MB/s | 202 kB 00:00 2023-07-20T01:12:41.325 INFO:teuthology.orchestra.run.smithi183.stdout:(97/131): openblas-0.3.15-3.el8.x86_64.rpm 22 MB/s | 4.7 MB 00:00 2023-07-20T01:12:41.350 INFO:teuthology.orchestra.run.smithi183.stdout:(98/131): librdkafka-0.11.4-3.el8.x86_64.rpm 3.2 MB/s | 354 kB 00:00 2023-07-20T01:12:41.493 INFO:teuthology.orchestra.run.smithi183.stdout:(99/131): postgresql-contrib-10.21-2.module+el8 5.6 MB/s | 808 kB 00:00 2023-07-20T01:12:41.559 INFO:teuthology.orchestra.run.smithi183.stdout:(100/131): postgresql-server-10.21-2.module+el8 22 MB/s | 5.1 MB 00:00 2023-07-20T01:12:41.601 INFO:teuthology.orchestra.run.smithi183.stdout:(101/131): postgresql-10.21-2.module+el8.6.0+15 14 MB/s | 1.5 MB 00:00 2023-07-20T01:12:41.618 INFO:teuthology.orchestra.run.smithi183.stdout:(102/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 802 kB/s | 47 kB 00:00 2023-07-20T01:12:41.643 INFO:teuthology.orchestra.run.smithi183.stdout:(103/131): libselinux-devel-2.9-5.el8.x86_64.rp 4.7 MB/s | 200 kB 00:00 2023-07-20T01:12:41.660 INFO:teuthology.orchestra.run.smithi183.stdout:(104/131): python3-jwt-1.6.1-2.el8.noarch.rpm 1.0 MB/s | 43 kB 00:00 2023-07-20T01:12:41.686 INFO:teuthology.orchestra.run.smithi183.stdout:(105/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 5.1 MB/s | 220 kB 00:00 2023-07-20T01:12:41.703 INFO:teuthology.orchestra.run.smithi183.stdout:(106/131): keyutils-libs-devel-1.5.10-9.el8.x86 1.1 MB/s | 48 kB 00:00 2023-07-20T01:12:41.752 INFO:teuthology.orchestra.run.smithi183.stdout:(107/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.3 MB/s | 87 kB 00:00 2023-07-20T01:12:41.770 INFO:teuthology.orchestra.run.smithi183.stdout:(108/131): python3-requests-oauthlib-1.0.0-1.el 637 kB/s | 43 kB 00:00 2023-07-20T01:12:41.795 INFO:teuthology.orchestra.run.smithi183.stdout:(109/131): python3-pycparser-2.14-14.el8.noarch 2.5 MB/s | 109 kB 00:00 2023-07-20T01:12:41.821 INFO:teuthology.orchestra.run.smithi183.stdout:(110/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 4.5 MB/s | 229 kB 00:00 2023-07-20T01:12:41.854 INFO:teuthology.orchestra.run.smithi183.stdout:(111/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 9.3 MB/s | 560 kB 00:00 2023-07-20T01:12:41.880 INFO:teuthology.orchestra.run.smithi183.stdout:(112/131): libaio-devel-0.3.112-1.el8.x86_64.rp 318 kB/s | 19 kB 00:00 2023-07-20T01:12:41.905 INFO:teuthology.orchestra.run.smithi183.stdout:(113/131): python3-oauthlib-2.1.0-1.el8.noarch. 3.0 MB/s | 155 kB 00:00 2023-07-20T01:12:41.939 INFO:teuthology.orchestra.run.smithi183.stdout:(114/131): python3-cryptography-3.2.1-5.el8.x86 9.3 MB/s | 559 kB 00:00 2023-07-20T01:12:41.963 INFO:teuthology.orchestra.run.smithi183.stdout:(115/131): libverto-devel-0.3.0-5.el8.x86_64.rp 306 kB/s | 18 kB 00:00 2023-07-20T01:12:41.997 INFO:teuthology.orchestra.run.smithi183.stdout:(116/131): pcre2-devel-10.32-2.el8.x86_64.rpm 10 MB/s | 605 kB 00:00 2023-07-20T01:12:42.023 INFO:teuthology.orchestra.run.smithi183.stdout:(117/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 4.0 MB/s | 238 kB 00:00 2023-07-20T01:12:42.056 INFO:teuthology.orchestra.run.smithi183.stdout:(118/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 2.1 MB/s | 72 kB 00:00 2023-07-20T01:12:42.106 INFO:teuthology.orchestra.run.smithi183.stdout:(119/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 21 MB/s | 2.3 MB 00:00 2023-07-20T01:12:42.257 INFO:teuthology.orchestra.run.smithi183.stdout:(120/131): fuse-2.9.7-15.el8.x86_64.rpm 413 kB/s | 83 kB 00:00 2023-07-20T01:12:42.549 INFO:teuthology.orchestra.run.smithi183.stdout:(121/131): elfutils-libelf-devel-0.186-1.el8.x8 135 kB/s | 60 kB 00:00 2023-07-20T01:12:42.574 INFO:teuthology.orchestra.run.smithi183.stdout:(122/131): libuuid-devel-2.32.1-35.el8.x86_64.r 310 kB/s | 98 kB 00:00 2023-07-20T01:12:42.600 INFO:teuthology.orchestra.run.smithi183.stdout:(123/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 1.1 MB/s | 58 kB 00:00 2023-07-20T01:12:42.616 INFO:teuthology.orchestra.run.smithi183.stdout:(124/131): libcom_err-devel-1.45.6-4.el8.x86_64 925 kB/s | 39 kB 00:00 2023-07-20T01:12:42.650 INFO:teuthology.orchestra.run.smithi183.stdout:(125/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 13 MB/s | 644 kB 00:00 2023-07-20T01:12:42.675 INFO:teuthology.orchestra.run.smithi183.stdout:(126/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 2.9 MB/s | 171 kB 00:00 2023-07-20T01:12:42.700 INFO:teuthology.orchestra.run.smithi183.stdout:(127/131): NetworkManager-initscripts-updown-1. 2.7 MB/s | 138 kB 00:00 2023-07-20T01:12:43.184 INFO:teuthology.orchestra.run.smithi183.stdout:(128/131): ceph-test-18.0.0-5040.g1b0b2eb0.el8. 3.8 MB/s | 48 MB 00:12 2023-07-20T01:12:43.234 INFO:teuthology.orchestra.run.smithi183.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 16 MB/s | 798 kB 00:00 2023-07-20T01:12:43.951 INFO:teuthology.orchestra.run.smithi183.stdout:(130/131): librbd1-18.0.0-5040.g1b0b2eb0.el8.x8 2.9 MB/s | 3.6 MB 00:01 2023-07-20T01:12:44.243 INFO:teuthology.orchestra.run.smithi183.stdout:(131/131): librados2-18.0.0-5040.g1b0b2eb0.el8. 2.3 MB/s | 3.7 MB 00:01 2023-07-20T01:12:44.246 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-07-20T01:12:44.246 INFO:teuthology.orchestra.run.smithi183.stdout:Total 10 MB/s | 206 MB 00:20 2023-07-20T01:12:45.096 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-07-20T01:12:45.217 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-07-20T01:12:45.217 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-07-20T01:12:46.339 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-07-20T01:12:46.343 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-07-20T01:12:47.128 INFO:teuthology.orchestra.run.smithi060.stdout:(8/131): ceph-common-18.0.0-5040.g1b0b2eb0.el8. 1.3 MB/s | 22 MB 00:16 2023-07-20T01:12:47.195 INFO:teuthology.orchestra.run.smithi060.stdout:(9/131): ceph-selinux-18.0.0-5040.g1b0b2eb0.el8 366 kB/s | 24 kB 00:00 2023-07-20T01:12:48.262 INFO:teuthology.orchestra.run.smithi060.stdout:(10/131): ceph-radosgw-18.0.0-5040.g1b0b2eb0.el 1.3 MB/s | 14 MB 00:10 2023-07-20T01:12:48.454 INFO:teuthology.orchestra.run.smithi060.stdout:(11/131): ceph-osd-18.0.0-5040.g1b0b2eb0.el8.x8 1.5 MB/s | 18 MB 00:12 2023-07-20T01:12:48.555 INFO:teuthology.orchestra.run.smithi060.stdout:(12/131): libcephfs-devel-18.0.0-5040.g1b0b2eb0 310 kB/s | 31 kB 00:00 2023-07-20T01:12:48.613 INFO:teuthology.orchestra.run.smithi060.stdout:(13/131): cephfs-mirror-18.0.0-5040.g1b0b2eb0.e 742 kB/s | 260 kB 00:00 2023-07-20T01:12:48.780 INFO:teuthology.orchestra.run.smithi060.stdout:(14/131): libcephsqlite-18.0.0-5040.g1b0b2eb0.e 834 kB/s | 139 kB 00:00 2023-07-20T01:12:48.939 INFO:teuthology.orchestra.run.smithi060.stdout:(15/131): librados-devel-18.0.0-5040.g1b0b2eb0. 877 kB/s | 139 kB 00:00 2023-07-20T01:12:49.156 INFO:teuthology.orchestra.run.smithi060.stdout:(16/131): libcephfs2-18.0.0-5040.g1b0b2eb0.el8. 1.3 MB/s | 771 kB 00:00 2023-07-20T01:12:49.281 INFO:teuthology.orchestra.run.smithi060.stdout:(17/131): libradosstriper1-18.0.0-5040.g1b0b2eb 1.3 MB/s | 472 kB 00:00 2023-07-20T01:12:49.364 INFO:teuthology.orchestra.run.smithi060.stdout:(18/131): python3-ceph-argparse-18.0.0-5040.g1b 537 kB/s | 45 kB 00:00 2023-07-20T01:12:49.506 INFO:teuthology.orchestra.run.smithi060.stdout:(19/131): python3-ceph-common-18.0.0-5040.g1b0b 865 kB/s | 122 kB 00:00 2023-07-20T01:12:49.707 INFO:teuthology.orchestra.run.smithi060.stdout:(20/131): python3-cephfs-18.0.0-5040.g1b0b2eb0. 985 kB/s | 197 kB 00:00 2023-07-20T01:12:50.049 INFO:teuthology.orchestra.run.smithi060.stdout:(21/131): python3-rados-18.0.0-5040.g1b0b2eb0.e 1.1 MB/s | 370 kB 00:00 2023-07-20T01:12:50.349 INFO:teuthology.orchestra.run.smithi060.stdout:(22/131): python3-rbd-18.0.0-5040.g1b0b2eb0.el8 1.2 MB/s | 363 kB 00:00 2023-07-20T01:12:50.515 INFO:teuthology.orchestra.run.smithi060.stdout:(23/131): python3-rgw-18.0.0-5040.g1b0b2eb0.el8 640 kB/s | 107 kB 00:00 2023-07-20T01:12:50.616 INFO:teuthology.orchestra.run.smithi060.stdout:(24/131): rbd-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 938 kB/s | 94 kB 00:00 2023-07-20T01:12:51.178 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-07-20T01:12:51.365 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-20T01:12:51.579 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-20T01:12:51.807 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-07-20T01:12:51.975 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-20T01:12:52.093 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-20T01:12:52.300 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-20T01:12:52.710 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-20T01:12:53.075 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T01:12:53.473 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T01:12:53.666 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T01:12:53.826 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T01:12:54.008 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T01:12:54.106 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T01:12:54.301 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 9/134 2023-07-20T01:12:54.520 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-20T01:12:54.702 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-20T01:12:55.106 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-20T01:12:55.257 INFO:teuthology.orchestra.run.smithi060.stdout:(25/131): rbd-mirror-18.0.0-5040.g1b0b2eb0.el8. 779 kB/s | 3.5 MB 00:04 2023-07-20T01:12:55.303 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-20T01:12:55.441 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T01:12:55.528 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T01:12:55.682 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T01:12:55.766 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T01:12:55.907 INFO:teuthology.orchestra.run.smithi060.stdout:(26/131): rbd-nbd-18.0.0-5040.g1b0b2eb0.el8.x86 275 kB/s | 179 kB 00:00 2023-07-20T01:12:55.932 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-20T01:12:56.073 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-20T01:12:56.166 INFO:teuthology.orchestra.run.smithi060.stdout:(27/131): ceph-grafana-dashboards-18.0.0-5040.g 89 kB/s | 23 kB 00:00 2023-07-20T01:12:56.232 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-20T01:12:56.458 INFO:teuthology.orchestra.run.smithi060.stdout:(28/131): librgw2-18.0.0-5040.g1b0b2eb0.el8.x86 950 kB/s | 6.8 MB 00:07 2023-07-20T01:12:56.650 INFO:teuthology.orchestra.run.smithi060.stdout:(29/131): ceph-mgr-cephadm-18.0.0-5040.g1b0b2eb 267 kB/s | 129 kB 00:00 2023-07-20T01:12:56.849 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-20T01:12:57.083 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-20T01:12:57.683 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-20T01:12:57.865 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-07-20T01:12:58.587 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-07-20T01:12:58.855 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-07-20T01:12:59.091 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-07-20T01:12:59.175 INFO:teuthology.orchestra.run.smithi060.stdout:(30/131): ceph-mgr-dashboard-18.0.0-5040.g1b0b2 646 kB/s | 1.7 MB 00:02 2023-07-20T01:12:59.352 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-07-20T01:12:59.534 INFO:teuthology.orchestra.run.smithi060.stdout:(31/131): ceph-mgr-modules-core-18.0.0-5040.g1b 672 kB/s | 241 kB 00:00 2023-07-20T01:12:59.546 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-07-20T01:12:59.659 INFO:teuthology.orchestra.run.smithi060.stdout:(32/131): ceph-mgr-rook-18.0.0-5040.g1b0b2eb0.e 380 kB/s | 47 kB 00:00 2023-07-20T01:12:59.771 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-07-20T01:12:59.801 INFO:teuthology.orchestra.run.smithi060.stdout:(33/131): ceph-prometheus-alerts-18.0.0-5040.g1 102 kB/s | 15 kB 00:00 2023-07-20T01:13:00.008 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-20T01:13:00.172 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 26/134 2023-07-20T01:13:00.284 INFO:teuthology.orchestra.run.smithi060.stdout:(34/131): ceph-volume-18.0.0-5040.g1b0b2eb0.el8 534 kB/s | 258 kB 00:00 2023-07-20T01:13:00.335 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 27/134 2023-07-20T01:13:00.510 INFO:teuthology.orchestra.run.smithi060.stdout:(35/131): cephadm-18.0.0-5040.g1b0b2eb0.el8.noa 735 kB/s | 165 kB 00:00 2023-07-20T01:13:00.643 INFO:teuthology.orchestra.run.smithi060.stdout:(36/131): cephfs-top-18.0.0-5040.g1b0b2eb0.el8. 135 kB/s | 18 kB 00:00 2023-07-20T01:13:01.245 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 28/134 2023-07-20T01:13:01.606 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-07-20T01:13:01.867 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-07-20T01:13:01.960 INFO:teuthology.orchestra.run.smithi060.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 233 kB/s | 306 kB 00:01 2023-07-20T01:13:02.130 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-07-20T01:13:02.177 INFO:teuthology.orchestra.run.smithi060.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 271 kB/s | 59 kB 00:00 2023-07-20T01:13:02.331 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-20T01:13:02.419 INFO:teuthology.orchestra.run.smithi060.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 310 kB/s | 75 kB 00:00 2023-07-20T01:13:03.786 INFO:teuthology.orchestra.run.smithi060.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 348 kB/s | 476 kB 00:01 2023-07-20T01:13:04.070 INFO:teuthology.orchestra.run.smithi060.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 154 kB/s | 44 kB 00:00 2023-07-20T01:13:04.086 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-20T01:13:04.358 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-07-20T01:13:04.534 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-20T01:13:04.732 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-20T01:13:04.862 INFO:teuthology.orchestra.run.smithi060.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 234 kB/s | 185 kB 00:00 2023-07-20T01:13:04.936 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 36/134 2023-07-20T01:13:04.971 INFO:teuthology.orchestra.run.smithi060.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 304 kB/s | 33 kB 00:00 2023-07-20T01:13:05.096 INFO:teuthology.orchestra.run.smithi060.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 130 kB/s | 16 kB 00:00 2023-07-20T01:13:05.118 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 37/134 2023-07-20T01:13:05.271 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-07-20T01:13:05.423 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-07-20T01:13:05.600 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-07-20T01:13:05.776 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-07-20T01:13:05.979 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-07-20T01:13:06.054 INFO:teuthology.orchestra.run.smithi060.stdout:(45/131): python3-cheroot-8.5.2-1.el8.noarch.rp 180 kB/s | 173 kB 00:00 2023-07-20T01:13:06.161 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-07-20T01:13:06.544 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-07-20T01:13:06.817 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-07-20T01:13:07.015 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-07-20T01:13:07.220 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-07-20T01:13:07.485 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-07-20T01:13:07.781 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-07-20T01:13:08.013 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-07-20T01:13:08.187 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-07-20T01:13:08.376 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-07-20T01:13:08.559 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-07-20T01:13:08.742 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-07-20T01:13:09.022 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-07-20T01:13:09.267 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-07-20T01:13:09.526 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-07-20T01:13:09.745 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-07-20T01:13:09.912 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-07-20T01:13:10.098 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-07-20T01:13:10.271 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-07-20T01:13:10.342 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-07-20T01:13:10.599 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-07-20T01:13:10.799 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-07-20T01:13:10.954 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-20T01:13:11.020 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-20T01:13:11.250 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-07-20T01:13:11.447 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-07-20T01:13:11.595 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-20T01:13:11.773 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-20T01:13:12.020 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-20T01:13:12.286 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-20T01:13:12.425 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-20T01:13:12.590 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-20T01:13:12.779 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-20T01:13:12.982 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-20T01:13:13.164 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-20T01:13:13.349 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-20T01:13:13.379 INFO:teuthology.orchestra.run.smithi060.stdout:(46/131): python3-cherrypy-18.4.0-1.el8.noarch. 52 kB/s | 384 kB 00:07 2023-07-20T01:13:13.485 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-20T01:13:13.722 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-20T01:13:14.162 INFO:teuthology.orchestra.run.smithi060.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 104 kB/s | 81 kB 00:00 2023-07-20T01:13:14.212 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-20T01:13:14.296 INFO:teuthology.orchestra.run.smithi060.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 83 kB/s | 11 kB 00:00 2023-07-20T01:13:14.399 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-20T01:13:14.438 INFO:teuthology.orchestra.run.smithi060.stdout:(49/131): python3-jaraco-functools-2.0-4.el8.no 124 kB/s | 18 kB 00:00 2023-07-20T01:13:14.635 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-20T01:13:14.853 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-20T01:13:15.027 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-20T01:13:15.253 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-20T01:13:16.379 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-20T01:13:16.699 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-20T01:13:16.928 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-20T01:13:17.229 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-20T01:13:17.489 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-20T01:13:17.713 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-20T01:13:18.121 INFO:teuthology.orchestra.run.smithi060.stdout:(50/131): python3-kubernetes-11.0.0-6.el8.noarc 324 kB/s | 1.2 MB 00:03 2023-07-20T01:13:18.280 INFO:teuthology.orchestra.run.smithi060.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 308 kB/s | 49 kB 00:00 2023-07-20T01:13:18.414 INFO:teuthology.orchestra.run.smithi060.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 440 kB/s | 59 kB 00:00 2023-07-20T01:13:18.489 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-20T01:13:18.573 INFO:teuthology.orchestra.run.smithi060.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 381 kB/s | 60 kB 00:00 2023-07-20T01:13:18.662 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T01:13:18.757 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T01:13:18.920 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 93/134 2023-07-20T01:13:19.173 INFO:teuthology.orchestra.run.smithi060.stdout:(54/131): python3-pecan-1.3.2-9.el8.noarch.rpm 471 kB/s | 283 kB 00:00 2023-07-20T01:13:19.339 INFO:teuthology.orchestra.run.smithi060.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 93 kB/s | 16 kB 00:00 2023-07-20T01:13:19.590 INFO:teuthology.orchestra.run.smithi060.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 134 kB/s | 34 kB 00:00 2023-07-20T01:13:19.798 INFO:teuthology.orchestra.run.smithi060.stdout:(57/131): ceph-mgr-diskprediction-local-18.0.0- 328 kB/s | 7.4 MB 00:23 2023-07-20T01:13:20.057 INFO:teuthology.orchestra.run.smithi060.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 421 kB/s | 196 kB 00:00 2023-07-20T01:13:20.216 INFO:teuthology.orchestra.run.smithi060.stdout:(59/131): python3-simplegeneric-0.8.1-17.el8.no 118 kB/s | 19 kB 00:00 2023-07-20T01:13:20.382 INFO:teuthology.orchestra.run.smithi060.stdout:(60/131): python3-singledispatch-3.4.0.3-18.el8 144 kB/s | 24 kB 00:00 2023-07-20T01:13:20.483 INFO:teuthology.orchestra.run.smithi060.stdout:(61/131): python3-rsa-4.8-1.el8.noarch.rpm 90 kB/s | 61 kB 00:00 2023-07-20T01:13:20.534 INFO:teuthology.orchestra.run.smithi060.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 194 kB/s | 29 kB 00:00 2023-07-20T01:13:20.673 INFO:teuthology.orchestra.run.smithi060.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 145 kB/s | 27 kB 00:00 2023-07-20T01:13:21.076 INFO:teuthology.orchestra.run.smithi060.stdout:(64/131): python3-waitress-1.4.3-1.el8.noarch.r 471 kB/s | 255 kB 00:00 2023-07-20T01:13:21.242 INFO:teuthology.orchestra.run.smithi060.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 368 kB/s | 61 kB 00:00 2023-07-20T01:13:21.501 INFO:teuthology.orchestra.run.smithi060.stdout:(66/131): python3-webob-1.8.5-1.el8.1.noarch.rp 303 kB/s | 251 kB 00:00 2023-07-20T01:13:21.560 INFO:teuthology.orchestra.run.smithi060.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 269 kB/s | 85 kB 00:00 2023-07-20T01:13:21.668 INFO:teuthology.orchestra.run.smithi060.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 137 kB/s | 23 kB 00:00 2023-07-20T01:13:21.694 INFO:teuthology.orchestra.run.smithi060.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-07-20T01:13:21.744 INFO:teuthology.orchestra.run.smithi060.stdout:(70/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 934 kB/s | 47 kB 00:00 2023-07-20T01:13:21.786 INFO:teuthology.orchestra.run.smithi060.stdout:(71/131): libselinux-devel-2.9-5.el8.x86_64.rpm 4.7 MB/s | 200 kB 00:00 2023-07-20T01:13:21.820 INFO:teuthology.orchestra.run.smithi060.stdout:(72/131): python3-jwt-1.6.1-2.el8.noarch.rpm 1.3 MB/s | 43 kB 00:00 2023-07-20T01:13:21.861 INFO:teuthology.orchestra.run.smithi060.stdout:(73/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 5.2 MB/s | 220 kB 00:00 2023-07-20T01:13:21.895 INFO:teuthology.orchestra.run.smithi060.stdout:(74/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 1.4 MB/s | 48 kB 00:00 2023-07-20T01:13:21.937 INFO:teuthology.orchestra.run.smithi060.stdout:(75/131): libsepol-devel-2.9-3.el8.x86_64.rpm 2.0 MB/s | 87 kB 00:00 2023-07-20T01:13:21.963 INFO:teuthology.orchestra.run.smithi060.stdout:(76/131): python3-requests-oauthlib-1.0.0-1.el8 1.7 MB/s | 43 kB 00:00 2023-07-20T01:13:22.005 INFO:teuthology.orchestra.run.smithi060.stdout:(77/131): python3-pycparser-2.14-14.el8.noarch. 2.6 MB/s | 109 kB 00:00 2023-07-20T01:13:22.064 INFO:teuthology.orchestra.run.smithi060.stdout:(78/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 3.8 MB/s | 229 kB 00:00 2023-07-20T01:13:22.122 INFO:teuthology.orchestra.run.smithi060.stdout:(79/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 9.6 MB/s | 560 kB 00:00 2023-07-20T01:13:22.155 INFO:teuthology.orchestra.run.smithi060.stdout:(80/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 563 kB/s | 19 kB 00:00 2023-07-20T01:13:22.197 INFO:teuthology.orchestra.run.smithi060.stdout:(81/131): python3-oauthlib-2.1.0-1.el8.noarch.r 3.6 MB/s | 155 kB 00:00 2023-07-20T01:13:22.248 INFO:teuthology.orchestra.run.smithi060.stdout:(82/131): python3-cryptography-3.2.1-5.el8.x86_ 11 MB/s | 559 kB 00:00 2023-07-20T01:13:22.281 INFO:teuthology.orchestra.run.smithi060.stdout:(83/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 537 kB/s | 18 kB 00:00 2023-07-20T01:13:22.331 INFO:teuthology.orchestra.run.smithi060.stdout:(84/131): pcre2-devel-10.32-2.el8.x86_64.rpm 12 MB/s | 605 kB 00:00 2023-07-20T01:13:22.373 INFO:teuthology.orchestra.run.smithi060.stdout:(85/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 5.6 MB/s | 238 kB 00:00 2023-07-20T01:13:22.457 INFO:teuthology.orchestra.run.smithi060.stdout:(86/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 28 MB/s | 2.3 MB 00:00 2023-07-20T01:13:22.491 INFO:teuthology.orchestra.run.smithi060.stdout:(87/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 2.1 MB/s | 72 kB 00:00 2023-07-20T01:13:22.533 INFO:teuthology.orchestra.run.smithi060.stdout:(88/131): fuse-2.9.7-15.el8.x86_64.rpm 1.9 MB/s | 83 kB 00:00 2023-07-20T01:13:22.552 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T01:13:22.575 INFO:teuthology.orchestra.run.smithi060.stdout:(89/131): elfutils-libelf-devel-0.186-1.el8.x86 1.4 MB/s | 60 kB 00:00 2023-07-20T01:13:22.625 INFO:teuthology.orchestra.run.smithi060.stdout:(90/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 1.9 MB/s | 98 kB 00:00 2023-07-20T01:13:22.667 INFO:teuthology.orchestra.run.smithi060.stdout:(91/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 1.4 MB/s | 58 kB 00:00 2023-07-20T01:13:22.709 INFO:teuthology.orchestra.run.smithi060.stdout:(92/131): libcom_err-devel-1.45.6-4.el8.x86_64. 930 kB/s | 39 kB 00:00 2023-07-20T01:13:22.784 INFO:teuthology.orchestra.run.smithi060.stdout:(93/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 8.4 MB/s | 644 kB 00:00 2023-07-20T01:13:22.795 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T01:13:22.835 INFO:teuthology.orchestra.run.smithi060.stdout:(94/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 3.3 MB/s | 171 kB 00:00 2023-07-20T01:13:22.877 INFO:teuthology.orchestra.run.smithi060.stdout:(95/131): NetworkManager-initscripts-updown-1.3 3.2 MB/s | 138 kB 00:00 2023-07-20T01:13:23.177 INFO:teuthology.orchestra.run.smithi060.stdout:(96/131): python3-scipy-1.0.0-21.module+el8.5.0 46 MB/s | 14 MB 00:00 2023-07-20T01:13:23.219 INFO:teuthology.orchestra.run.smithi060.stdout:(97/131): python3-pyasn1-modules-0.3.7-6.el8.no 2.6 MB/s | 110 kB 00:00 2023-07-20T01:13:23.261 INFO:teuthology.orchestra.run.smithi060.stdout:(98/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.3 MB/s | 54 kB 00:00 2023-07-20T01:13:23.303 INFO:teuthology.orchestra.run.smithi060.stdout:(99/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.1 MB/s | 259 kB 00:00 2023-07-20T01:13:23.345 INFO:teuthology.orchestra.run.smithi060.stdout:(100/131): socat-1.7.4.1-1.el8.x86_64.rpm 7.5 MB/s | 323 kB 00:00 2023-07-20T01:13:23.383 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T01:13:23.387 INFO:teuthology.orchestra.run.smithi060.stdout:(101/131): python3-markupsafe-0.23-19.el8.x86_6 935 kB/s | 39 kB 00:00 2023-07-20T01:13:23.437 INFO:teuthology.orchestra.run.smithi060.stdout:(102/131): bison-3.0.4-10.el8.x86_64.rpm 13 MB/s | 688 kB 00:00 2023-07-20T01:13:23.496 INFO:teuthology.orchestra.run.smithi060.stdout:(103/131): python3-jinja2-2.10.1-3.el8.noarch.r 9.0 MB/s | 538 kB 00:00 2023-07-20T01:13:23.538 INFO:teuthology.orchestra.run.smithi060.stdout:(104/131): python3-mako-1.0.6-13.el8.noarch.rpm 3.7 MB/s | 157 kB 00:00 2023-07-20T01:13:23.580 INFO:teuthology.orchestra.run.smithi060.stdout:(105/131): oniguruma-6.8.2-2.el8.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2023-07-20T01:13:23.630 INFO:teuthology.orchestra.run.smithi060.stdout:(106/131): automake-1.16.1-7.el8.noarch.rpm 14 MB/s | 713 kB 00:00 2023-07-20T01:13:23.656 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T01:13:23.739 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T01:13:23.764 INFO:teuthology.orchestra.run.smithi060.stdout:(107/131): python3-babel-2.5.1-7.el8.noarch.rpm 36 MB/s | 4.8 MB 00:00 2023-07-20T01:13:23.789 INFO:teuthology.orchestra.run.smithi060.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 1.0 MB/s | 27 kB 00:00 2023-07-20T01:13:23.831 INFO:teuthology.orchestra.run.smithi060.stdout:(109/131): flex-2.6.1-9.el8.x86_64.rpm 7.5 MB/s | 320 kB 00:00 2023-07-20T01:13:23.864 INFO:teuthology.orchestra.run.smithi060.stdout:(110/131): python3-prettytable-0.7.2-14.el8.noa 1.3 MB/s | 44 kB 00:00 2023-07-20T01:13:23.886 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T01:13:23.906 INFO:teuthology.orchestra.run.smithi060.stdout:(111/131): thrift-0.13.0-2.el8.x86_64.rpm 757 kB/s | 1.7 MB 00:02 2023-07-20T01:13:23.974 INFO:teuthology.orchestra.run.smithi060.stdout:(112/131): python3-werkzeug-0.12.2-4.el8.noarch 6.7 MB/s | 457 kB 00:00 2023-07-20T01:13:24.041 INFO:teuthology.orchestra.run.smithi060.stdout:(113/131): python3-numpy-1.14.3-10.el8.x86_64.r 21 MB/s | 3.7 MB 00:00 2023-07-20T01:13:24.066 INFO:teuthology.orchestra.run.smithi060.stdout:(114/131): python3-numpy-f2py-1.14.3-10.el8.x86 2.4 MB/s | 225 kB 00:00 2023-07-20T01:13:24.091 INFO:teuthology.orchestra.run.smithi060.stdout:(115/131): uuid-1.6.2-43.el8.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2023-07-20T01:13:24.108 INFO:teuthology.orchestra.run.smithi060.stdout:(116/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 2.9 MB/s | 126 kB 00:00 2023-07-20T01:13:24.142 INFO:teuthology.orchestra.run.smithi060.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 397 kB/s | 20 kB 00:00 2023-07-20T01:13:24.191 INFO:teuthology.orchestra.run.smithi060.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 1.2 MB/s | 103 kB 00:00 2023-07-20T01:13:24.234 INFO:teuthology.orchestra.run.smithi060.stdout:(119/131): libtool-2.4.6-25.el8.x86_64.rpm 7.5 MB/s | 709 kB 00:00 2023-07-20T01:13:24.268 INFO:teuthology.orchestra.run.smithi060.stdout:(120/131): libpq-13.5-1.el8.x86_64.rpm 5.8 MB/s | 198 kB 00:00 2023-07-20T01:13:24.385 INFO:teuthology.orchestra.run.smithi060.stdout:(121/131): openblas-threads-0.3.15-3.el8.x86_64 25 MB/s | 4.9 MB 00:00 2023-07-20T01:13:24.427 INFO:teuthology.orchestra.run.smithi060.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 4.7 MB/s | 202 kB 00:00 2023-07-20T01:13:24.493 INFO:teuthology.orchestra.run.smithi060.stdout:(123/131): openblas-0.3.15-3.el8.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2023-07-20T01:13:24.519 INFO:teuthology.orchestra.run.smithi060.stdout:(124/131): librdkafka-0.11.4-3.el8.x86_64.rpm 3.8 MB/s | 354 kB 00:00 2023-07-20T01:13:24.569 INFO:teuthology.orchestra.run.smithi060.stdout:(125/131): postgresql-contrib-10.21-2.module+el 16 MB/s | 808 kB 00:00 2023-07-20T01:13:24.636 INFO:teuthology.orchestra.run.smithi060.stdout:(126/131): postgresql-10.21-2.module+el8.6.0+15 23 MB/s | 1.5 MB 00:00 2023-07-20T01:13:24.711 INFO:teuthology.orchestra.run.smithi060.stdout:(127/131): postgresql-server-10.21-2.module+el8 23 MB/s | 5.1 MB 00:00 2023-07-20T01:13:34.277 INFO:teuthology.orchestra.run.smithi060.stdout:(128/131): librados2-18.0.0-5040.g1b0b2eb0.el8. 391 kB/s | 3.7 MB 00:09 2023-07-20T01:13:34.336 INFO:teuthology.orchestra.run.smithi060.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 13 MB/s | 798 kB 00:00 2023-07-20T01:13:39.511 INFO:teuthology.orchestra.run.smithi060.stdout:(130/131): librbd1-18.0.0-5040.g1b0b2eb0.el8.x8 251 kB/s | 3.6 MB 00:14 2023-07-20T01:13:42.762 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T01:13:42.763 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /sys 2023-07-20T01:13:42.763 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /proc 2023-07-20T01:13:42.763 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /dev 2023-07-20T01:13:42.763 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /run 2023-07-20T01:13:42.763 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /mnt 2023-07-20T01:13:42.763 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /var/tmp 2023-07-20T01:13:42.763 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /home 2023-07-20T01:13:42.764 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /tmp 2023-07-20T01:13:42.764 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /dev 2023-07-20T01:13:42.764 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-07-20T01:13:43.124 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T01:13:44.705 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T01:13:44.860 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T01:13:45.080 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T01:13:45.265 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T01:13:45.720 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T01:13:45.865 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T01:13:45.882 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T01:13:47.612 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T01:13:47.813 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T01:13:47.929 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 102/134 2023-07-20T01:13:48.120 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 103/134 2023-07-20T01:13:48.306 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T01:13:48.519 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T01:13:48.709 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T01:13:49.670 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T01:13:49.814 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T01:13:49.904 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T01:13:50.246 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 107/134 2023-07-20T01:13:50.473 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T01:13:50.508 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T01:13:50.697 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T01:13:50.708 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T01:13:52.014 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 110/134 2023-07-20T01:13:52.203 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T01:13:52.237 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T01:13:52.372 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T01:13:52.414 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T01:13:52.555 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T01:13:52.907 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T01:13:53.072 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T01:13:57.633 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T01:13:57.918 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 115/134 2023-07-20T01:13:58.062 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-07-20T01:13:58.252 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-07-20T01:13:59.112 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-07-20T01:13:59.435 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-07-20T01:13:59.598 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 119/134 2023-07-20T01:13:59.791 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-20T01:13:59.941 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 121/134 2023-07-20T01:14:00.088 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-20T01:14:02.192 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T01:14:02.437 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T01:14:02.455 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T01:14:02.629 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 124/134 2023-07-20T01:14:02.911 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 125/134 2023-07-20T01:14:03.210 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 126/134 2023-07-20T01:14:03.471 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-20T01:14:03.595 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-20T01:14:03.603 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-20T01:14:03.799 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-07-20T01:14:03.986 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-07-20T01:14:04.075 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-07-20T01:14:04.261 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-07-20T01:14:04.274 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-07-20T01:14:04.274 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T01:14:04.481 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T01:14:04.481 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T01:14:04.725 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T01:14:04.935 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T01:14:05.466 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T01:14:05.466 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 1/134 2023-07-20T01:14:05.466 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2/134 2023-07-20T01:14:05.466 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 3/134 2023-07-20T01:14:05.466 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 4/134 2023-07-20T01:14:05.466 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 5/134 2023-07-20T01:14:05.467 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T01:14:05.467 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T01:14:05.467 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T01:14:05.467 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 9/134 2023-07-20T01:14:05.467 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 10/134 2023-07-20T01:14:05.467 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 11/134 2023-07-20T01:14:05.467 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 12/134 2023-07-20T01:14:05.468 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 13/134 2023-07-20T01:14:05.468 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 14/134 2023-07-20T01:14:05.468 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 15/134 2023-07-20T01:14:05.468 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 16/134 2023-07-20T01:14:05.468 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 17/134 2023-07-20T01:14:05.468 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 18/134 2023-07-20T01:14:05.468 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 19/134 2023-07-20T01:14:05.469 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 20/134 2023-07-20T01:14:05.469 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 21/134 2023-07-20T01:14:05.469 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 22/134 2023-07-20T01:14:05.469 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 23/134 2023-07-20T01:14:05.469 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 24/134 2023-07-20T01:14:05.469 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 25/134 2023-07-20T01:14:05.469 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 26/134 2023-07-20T01:14:05.469 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 27/134 2023-07-20T01:14:05.476 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 28/134 2023-07-20T01:14:05.476 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 29/134 2023-07-20T01:14:05.477 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 30/134 2023-07-20T01:14:05.477 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 31/134 2023-07-20T01:14:05.477 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 32/134 2023-07-20T01:14:05.477 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 33/134 2023-07-20T01:14:05.477 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 34/134 2023-07-20T01:14:05.477 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 35/134 2023-07-20T01:14:05.477 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 36/134 2023-07-20T01:14:05.478 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 37/134 2023-07-20T01:14:05.478 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 38/134 2023-07-20T01:14:05.478 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-20T01:14:05.478 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-20T01:14:05.478 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-20T01:14:05.478 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-20T01:14:05.478 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-20T01:14:05.478 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-20T01:14:05.479 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-20T01:14:05.479 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-20T01:14:05.479 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-20T01:14:05.479 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-20T01:14:05.479 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-20T01:14:05.479 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-20T01:14:05.479 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-20T01:14:05.480 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-20T01:14:05.480 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-20T01:14:05.480 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-20T01:14:05.480 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-20T01:14:05.480 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-20T01:14:05.480 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-20T01:14:05.480 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-20T01:14:05.480 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-20T01:14:05.481 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-20T01:14:05.481 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-20T01:14:05.481 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-20T01:14:05.481 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-20T01:14:05.481 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-20T01:14:05.481 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-20T01:14:05.481 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-20T01:14:05.481 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-20T01:14:05.482 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-20T01:14:05.482 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-20T01:14:05.482 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-20T01:14:05.482 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-20T01:14:05.482 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-07-20T01:14:05.482 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-07-20T01:14:05.482 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-07-20T01:14:05.483 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-07-20T01:14:05.483 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-07-20T01:14:05.483 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-07-20T01:14:05.483 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-07-20T01:14:05.483 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-07-20T01:14:05.483 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-07-20T01:14:05.483 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-07-20T01:14:05.483 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-07-20T01:14:05.484 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-07-20T01:14:05.484 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-07-20T01:14:05.484 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-07-20T01:14:05.484 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-07-20T01:14:05.484 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-07-20T01:14:05.484 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-07-20T01:14:05.484 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-07-20T01:14:05.484 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-07-20T01:14:05.485 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-07-20T01:14:05.485 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-07-20T01:14:05.485 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-07-20T01:14:05.485 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-07-20T01:14:05.485 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-07-20T01:14:05.485 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-07-20T01:14:05.485 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-07-20T01:14:05.486 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-07-20T01:14:05.486 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-07-20T01:14:05.486 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-07-20T01:14:05.486 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-07-20T01:14:05.486 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-07-20T01:14:05.486 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-07-20T01:14:05.486 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-07-20T01:14:05.486 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-07-20T01:14:05.487 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-07-20T01:14:05.487 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-07-20T01:14:05.487 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-07-20T01:14:05.487 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-07-20T01:14:05.487 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-07-20T01:14:05.487 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-07-20T01:14:05.487 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-07-20T01:14:05.487 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-07-20T01:14:05.488 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-07-20T01:14:05.488 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-07-20T01:14:05.488 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-07-20T01:14:05.488 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-07-20T01:14:05.488 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-07-20T01:14:05.488 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-07-20T01:14:05.488 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-20T01:14:05.488 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-07-20T01:14:05.489 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-20T01:14:05.489 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-07-20T01:14:05.489 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-07-20T01:14:05.489 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-07-20T01:14:05.489 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-07-20T01:14:05.489 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-07-20T01:14:05.489 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-20T01:14:05.490 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 129/134 2023-07-20T01:14:05.490 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-20T01:14:05.491 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 131/134 2023-07-20T01:14:05.491 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T01:14:05.492 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-20T01:14:07.875 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T01:14:07.876 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:55 ago on Thu 20 Jul 2023 01:12:12 AM UTC. 2023-07-20T01:14:10.306 INFO:teuthology.orchestra.run.smithi183.stdout:Installed products updated. 2023-07-20T01:14:10.524 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-07-20T01:14:10.524 INFO:teuthology.orchestra.run.smithi183.stdout:Upgraded: 2023-07-20T01:14:10.525 INFO:teuthology.orchestra.run.smithi183.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-20T01:14:10.525 INFO:teuthology.orchestra.run.smithi183.stdout: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:14:10.525 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:14:10.525 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-07-20T01:14:10.525 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-20T01:14:10.525 INFO:teuthology.orchestra.run.smithi183.stdout: automake-1.16.1-7.el8.noarch 2023-07-20T01:14:10.525 INFO:teuthology.orchestra.run.smithi183.stdout: bison-3.0.4-10.el8.x86_64 2023-07-20T01:14:10.526 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:14:10.526 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:14:10.526 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:14:10.526 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:14:10.526 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:14:10.526 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:14:10.526 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:14:10.527 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:14:10.527 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:14:10.527 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:14:10.527 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:14:10.527 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:14:10.527 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:14:10.527 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:14:10.527 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:14:10.528 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:14:10.528 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:14:10.528 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:14:10.528 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:14:10.528 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:14:10.528 INFO:teuthology.orchestra.run.smithi183.stdout: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:14:10.528 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:14:10.528 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:14:10.529 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-20T01:14:10.529 INFO:teuthology.orchestra.run.smithi183.stdout: flex-2.6.1-9.el8.x86_64 2023-07-20T01:14:10.529 INFO:teuthology.orchestra.run.smithi183.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-20T01:14:10.529 INFO:teuthology.orchestra.run.smithi183.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-20T01:14:10.529 INFO:teuthology.orchestra.run.smithi183.stdout: jq-1.6-3.el8.x86_64 2023-07-20T01:14:10.529 INFO:teuthology.orchestra.run.smithi183.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-20T01:14:10.529 INFO:teuthology.orchestra.run.smithi183.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-20T01:14:10.530 INFO:teuthology.orchestra.run.smithi183.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-20T01:14:10.530 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:14:10.530 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:14:10.530 INFO:teuthology.orchestra.run.smithi183.stdout: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:14:10.530 INFO:teuthology.orchestra.run.smithi183.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-20T01:14:10.530 INFO:teuthology.orchestra.run.smithi183.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-20T01:14:10.530 INFO:teuthology.orchestra.run.smithi183.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-20T01:14:10.530 INFO:teuthology.orchestra.run.smithi183.stdout: libpq-13.5-1.el8.x86_64 2023-07-20T01:14:10.531 INFO:teuthology.orchestra.run.smithi183.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-20T01:14:10.531 INFO:teuthology.orchestra.run.smithi183.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-20T01:14:10.531 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:14:10.531 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:14:10.531 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-20T01:14:10.531 INFO:teuthology.orchestra.run.smithi183.stdout: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:14:10.531 INFO:teuthology.orchestra.run.smithi183.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-20T01:14:10.531 INFO:teuthology.orchestra.run.smithi183.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-20T01:14:10.532 INFO:teuthology.orchestra.run.smithi183.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-20T01:14:10.532 INFO:teuthology.orchestra.run.smithi183.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-20T01:14:10.532 INFO:teuthology.orchestra.run.smithi183.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-20T01:14:10.532 INFO:teuthology.orchestra.run.smithi183.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-20T01:14:10.532 INFO:teuthology.orchestra.run.smithi183.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-20T01:14:10.532 INFO:teuthology.orchestra.run.smithi183.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-20T01:14:10.532 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-20T01:14:10.533 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-20T01:14:10.533 INFO:teuthology.orchestra.run.smithi183.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-20T01:14:10.533 INFO:teuthology.orchestra.run.smithi183.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-20T01:14:10.533 INFO:teuthology.orchestra.run.smithi183.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-20T01:14:10.533 INFO:teuthology.orchestra.run.smithi183.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-20T01:14:10.533 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T01:14:10.533 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T01:14:10.533 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T01:14:10.534 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-20T01:14:10.534 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-20T01:14:10.534 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-20T01:14:10.534 INFO:teuthology.orchestra.run.smithi183.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-20T01:14:10.534 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-20T01:14:10.534 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:14:10.534 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:14:10.535 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:14:10.535 INFO:teuthology.orchestra.run.smithi183.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-20T01:14:10.535 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-20T01:14:10.535 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-20T01:14:10.535 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-20T01:14:10.535 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-20T01:14:10.535 INFO:teuthology.orchestra.run.smithi183.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-20T01:14:10.535 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-20T01:14:10.536 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-20T01:14:10.536 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-20T01:14:10.536 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-20T01:14:10.536 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-20T01:14:10.536 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-20T01:14:10.536 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-20T01:14:10.536 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-20T01:14:10.536 INFO:teuthology.orchestra.run.smithi183.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-20T01:14:10.537 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-20T01:14:10.537 INFO:teuthology.orchestra.run.smithi183.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-20T01:14:10.537 INFO:teuthology.orchestra.run.smithi183.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-20T01:14:10.537 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-20T01:14:10.537 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-20T01:14:10.537 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-20T01:14:10.537 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-20T01:14:10.538 INFO:teuthology.orchestra.run.smithi183.stdout: python3-portend-2.6-1.el8.noarch 2023-07-20T01:14:10.538 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-20T01:14:10.538 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-20T01:14:10.538 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-20T01:14:10.538 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-20T01:14:10.538 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-20T01:14:10.538 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-20T01:14:10.538 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:14:10.539 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:14:10.539 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-20T01:14:10.539 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-20T01:14:10.539 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:14:10.539 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-20T01:14:10.539 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-20T01:14:10.539 INFO:teuthology.orchestra.run.smithi183.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-20T01:14:10.539 INFO:teuthology.orchestra.run.smithi183.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-20T01:14:10.540 INFO:teuthology.orchestra.run.smithi183.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-20T01:14:10.540 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-20T01:14:10.540 INFO:teuthology.orchestra.run.smithi183.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-20T01:14:10.540 INFO:teuthology.orchestra.run.smithi183.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-20T01:14:10.540 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-20T01:14:10.540 INFO:teuthology.orchestra.run.smithi183.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-20T01:14:10.541 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-20T01:14:10.541 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-20T01:14:10.541 INFO:teuthology.orchestra.run.smithi183.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-20T01:14:10.541 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:14:10.541 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:14:10.541 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:14:10.541 INFO:teuthology.orchestra.run.smithi183.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-20T01:14:10.542 INFO:teuthology.orchestra.run.smithi183.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-20T01:14:10.542 INFO:teuthology.orchestra.run.smithi183.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-20T01:14:10.542 INFO:teuthology.orchestra.run.smithi183.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-20T01:14:10.542 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-20T01:14:10.542 INFO:teuthology.orchestra.run.smithi183.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-20T01:14:10.543 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-07-20T01:14:10.543 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-07-20T01:14:10.706 DEBUG:teuthology.parallel:result is None 2023-07-20T01:14:12.573 INFO:teuthology.orchestra.run.smithi060.stdout:(131/131): ceph-test-18.0.0-5040.g1b0b2eb0.el8. 575 kB/s | 48 MB 01:25 2023-07-20T01:14:12.577 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2023-07-20T01:14:12.577 INFO:teuthology.orchestra.run.smithi060.stdout:Total 2.0 MB/s | 206 MB 01:42 2023-07-20T01:14:13.381 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2023-07-20T01:14:13.514 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2023-07-20T01:14:13.514 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2023-07-20T01:14:14.691 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2023-07-20T01:14:14.695 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2023-07-20T01:14:19.743 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2023-07-20T01:14:19.955 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-20T01:14:20.140 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-20T01:14:20.423 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-20T01:14:20.574 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-20T01:14:20.798 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-20T01:14:21.229 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-20T01:14:21.423 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 5/134 2023-07-20T01:14:21.840 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 5/134 2023-07-20T01:14:22.032 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T01:14:22.160 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T01:14:22.448 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T01:14:22.546 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T01:14:22.745 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T01:14:22.908 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-07-20T01:14:23.143 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-20T01:14:23.386 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-20T01:14:23.782 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-20T01:14:24.002 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-20T01:14:24.148 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T01:14:24.261 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T01:14:24.406 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T01:14:24.559 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T01:14:24.790 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-07-20T01:14:24.963 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-07-20T01:14:25.140 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-07-20T01:14:25.272 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-20T01:14:25.419 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-20T01:14:25.564 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-20T01:14:26.206 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-20T01:14:26.348 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-20T01:14:26.961 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-20T01:14:27.114 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-07-20T01:14:27.853 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-07-20T01:14:28.087 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-07-20T01:14:28.273 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-07-20T01:14:28.511 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-20T01:14:28.662 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 26/134 2023-07-20T01:14:28.801 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 27/134 2023-07-20T01:14:29.042 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 28/134 2023-07-20T01:14:29.214 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-20T01:14:31.097 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-20T01:14:32.130 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-07-20T01:14:32.447 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-07-20T01:14:32.737 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-07-20T01:14:32.943 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-07-20T01:14:33.144 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-20T01:14:33.367 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-20T01:14:33.545 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 36/134 2023-07-20T01:14:33.704 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 37/134 2023-07-20T01:14:33.846 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-07-20T01:14:34.008 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-07-20T01:14:34.179 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-07-20T01:14:34.255 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-07-20T01:14:34.475 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-07-20T01:14:34.699 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-07-20T01:14:34.838 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-20T01:14:34.916 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-20T01:14:35.130 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-07-20T01:14:35.326 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-07-20T01:14:35.489 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-07-20T01:14:35.649 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-07-20T01:14:35.816 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-07-20T01:14:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-07-20T01:14:36.157 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-07-20T01:14:36.348 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-07-20T01:14:36.709 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-07-20T01:14:37.000 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-07-20T01:14:37.217 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-07-20T01:14:37.463 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-07-20T01:14:37.704 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-07-20T01:14:38.038 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-07-20T01:14:38.254 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-07-20T01:14:38.448 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-07-20T01:14:38.687 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-07-20T01:14:38.885 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-07-20T01:14:39.068 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-07-20T01:14:39.364 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-07-20T01:14:39.658 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-07-20T01:14:39.880 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-07-20T01:14:40.089 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-07-20T01:14:40.263 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-20T01:14:40.443 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-20T01:14:40.681 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-20T01:14:40.922 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-20T01:14:41.076 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-20T01:14:41.217 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-20T01:14:41.418 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-20T01:14:41.611 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-20T01:14:41.801 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-20T01:14:41.974 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-20T01:14:42.118 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-20T01:14:42.345 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-20T01:14:42.542 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-20T01:14:42.726 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-20T01:14:42.991 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-20T01:14:43.229 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-20T01:14:43.388 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-20T01:14:43.600 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-20T01:14:44.772 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-20T01:14:45.035 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-20T01:14:45.296 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-20T01:14:45.642 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-20T01:14:45.903 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-20T01:14:46.114 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-20T01:14:46.918 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-20T01:14:47.397 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T01:14:47.535 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T01:14:47.695 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 93/134 2023-07-20T01:14:51.601 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T01:14:51.803 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T01:14:52.419 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T01:14:52.761 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T01:14:53.291 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T01:14:53.469 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T01:15:13.177 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T01:15:13.177 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /sys 2023-07-20T01:15:13.177 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /proc 2023-07-20T01:15:13.177 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /dev 2023-07-20T01:15:13.177 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /run 2023-07-20T01:15:13.178 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /mnt 2023-07-20T01:15:13.178 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /var/tmp 2023-07-20T01:15:13.178 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /home 2023-07-20T01:15:13.178 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /tmp 2023-07-20T01:15:13.178 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /dev 2023-07-20T01:15:13.178 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:15:13.541 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T01:15:15.221 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T01:15:15.408 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T01:15:15.638 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T01:15:15.807 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T01:15:16.288 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T01:15:16.424 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T01:15:16.441 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T01:15:18.182 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T01:15:18.394 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T01:15:18.530 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 102/134 2023-07-20T01:15:18.716 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 103/134 2023-07-20T01:15:18.898 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T01:15:19.123 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T01:15:19.366 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T01:15:20.393 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T01:15:20.573 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T01:15:20.671 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T01:15:21.028 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 107/134 2023-07-20T01:15:21.283 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T01:15:21.320 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T01:15:21.481 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T01:15:21.492 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T01:15:22.883 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 110/134 2023-07-20T01:15:23.062 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T01:15:23.097 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T01:15:23.248 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T01:15:23.291 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T01:15:23.439 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T01:15:23.814 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T01:15:24.006 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T01:15:28.925 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T01:15:29.899 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 115/134 2023-07-20T01:15:30.183 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-07-20T01:15:30.359 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 117/134 2023-07-20T01:15:30.623 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-07-20T01:15:30.771 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-07-20T01:15:30.986 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-07-20T01:15:31.235 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-07-20T01:15:31.393 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 121/134 2023-07-20T01:15:31.531 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-20T01:15:33.786 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T01:15:34.055 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T01:15:34.072 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T01:15:34.264 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 124/134 2023-07-20T01:15:34.514 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 125/134 2023-07-20T01:15:34.852 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 126/134 2023-07-20T01:15:35.195 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-20T01:15:35.404 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-07-20T01:15:35.492 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-07-20T01:15:35.696 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-07-20T01:15:35.709 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-07-20T01:15:35.871 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-20T01:15:35.879 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-20T01:15:36.049 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-07-20T01:15:36.049 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T01:15:36.225 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T01:15:36.225 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T01:15:36.461 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T01:15:36.662 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T01:15:37.219 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T01:15:37.220 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 1/134 2023-07-20T01:15:37.220 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2/134 2023-07-20T01:15:37.220 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 3/134 2023-07-20T01:15:37.220 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 4/134 2023-07-20T01:15:37.220 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 5/134 2023-07-20T01:15:37.220 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T01:15:37.221 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T01:15:37.221 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T01:15:37.221 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 9/134 2023-07-20T01:15:37.221 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 10/134 2023-07-20T01:15:37.221 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 11/134 2023-07-20T01:15:37.221 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 12/134 2023-07-20T01:15:37.221 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 13/134 2023-07-20T01:15:37.222 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 14/134 2023-07-20T01:15:37.222 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 15/134 2023-07-20T01:15:37.222 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 16/134 2023-07-20T01:15:37.222 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 17/134 2023-07-20T01:15:37.222 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 18/134 2023-07-20T01:15:37.222 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 19/134 2023-07-20T01:15:37.222 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 20/134 2023-07-20T01:15:37.222 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 21/134 2023-07-20T01:15:37.223 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 22/134 2023-07-20T01:15:37.223 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 23/134 2023-07-20T01:15:37.223 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 24/134 2023-07-20T01:15:37.223 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 25/134 2023-07-20T01:15:37.223 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 26/134 2023-07-20T01:15:37.223 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 27/134 2023-07-20T01:15:37.223 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 28/134 2023-07-20T01:15:37.224 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 29/134 2023-07-20T01:15:37.224 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 30/134 2023-07-20T01:15:37.224 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 31/134 2023-07-20T01:15:37.230 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 32/134 2023-07-20T01:15:37.231 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 33/134 2023-07-20T01:15:37.231 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 34/134 2023-07-20T01:15:37.231 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 35/134 2023-07-20T01:15:37.231 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 36/134 2023-07-20T01:15:37.231 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 37/134 2023-07-20T01:15:37.231 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 38/134 2023-07-20T01:15:37.231 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-20T01:15:37.232 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-20T01:15:37.232 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-20T01:15:37.232 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-20T01:15:37.232 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-20T01:15:37.232 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-20T01:15:37.232 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-20T01:15:37.232 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-20T01:15:37.233 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-20T01:15:37.233 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-20T01:15:37.233 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-20T01:15:37.233 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-20T01:15:37.233 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-20T01:15:37.233 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-20T01:15:37.233 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-20T01:15:37.234 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-20T01:15:37.234 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-20T01:15:37.234 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-20T01:15:37.234 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-20T01:15:37.234 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-20T01:15:37.234 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-20T01:15:37.235 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-20T01:15:37.235 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-20T01:15:37.235 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-20T01:15:37.235 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-20T01:15:37.235 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-20T01:15:37.236 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-20T01:15:37.236 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-20T01:15:37.236 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-20T01:15:37.236 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-20T01:15:37.236 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-20T01:15:37.236 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-20T01:15:37.237 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-20T01:15:37.237 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-07-20T01:15:37.237 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-07-20T01:15:37.237 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-07-20T01:15:37.237 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-07-20T01:15:37.237 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-07-20T01:15:37.237 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-07-20T01:15:37.237 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-07-20T01:15:37.238 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-07-20T01:15:37.238 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-07-20T01:15:37.238 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-07-20T01:15:37.238 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-07-20T01:15:37.238 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-07-20T01:15:37.238 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-07-20T01:15:37.238 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-07-20T01:15:37.238 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-07-20T01:15:37.239 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-07-20T01:15:37.239 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-07-20T01:15:37.239 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-07-20T01:15:37.239 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-07-20T01:15:37.239 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-07-20T01:15:37.239 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-07-20T01:15:37.239 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-07-20T01:15:37.239 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-07-20T01:15:37.240 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-07-20T01:15:37.240 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-07-20T01:15:37.240 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-07-20T01:15:37.240 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-07-20T01:15:37.240 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-07-20T01:15:37.240 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-07-20T01:15:37.240 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-07-20T01:15:37.241 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-07-20T01:15:37.241 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-07-20T01:15:37.241 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-07-20T01:15:37.241 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-07-20T01:15:37.241 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-07-20T01:15:37.241 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-07-20T01:15:37.241 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-07-20T01:15:37.242 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-07-20T01:15:37.242 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-07-20T01:15:37.242 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-07-20T01:15:37.242 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-07-20T01:15:37.242 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-07-20T01:15:37.242 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-07-20T01:15:37.242 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-07-20T01:15:37.242 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-07-20T01:15:37.243 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-07-20T01:15:37.243 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-07-20T01:15:37.243 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-07-20T01:15:37.243 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-07-20T01:15:37.243 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-07-20T01:15:37.243 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-07-20T01:15:37.243 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-07-20T01:15:37.243 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-07-20T01:15:37.244 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-07-20T01:15:37.244 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-07-20T01:15:37.244 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-07-20T01:15:37.244 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-07-20T01:15:37.244 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 129/134 2023-07-20T01:15:37.244 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-20T01:15:37.245 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 131/134 2023-07-20T01:15:37.245 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T01:15:37.245 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-20T01:15:39.664 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T01:15:39.664 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:03:22 ago on Thu 20 Jul 2023 01:12:17 AM UTC. 2023-07-20T01:15:42.351 INFO:teuthology.orchestra.run.smithi060.stdout:Installed products updated. 2023-07-20T01:15:42.582 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:15:42.582 INFO:teuthology.orchestra.run.smithi060.stdout:Upgraded: 2023-07-20T01:15:42.583 INFO:teuthology.orchestra.run.smithi060.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-20T01:15:42.583 INFO:teuthology.orchestra.run.smithi060.stdout: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:15:42.583 INFO:teuthology.orchestra.run.smithi060.stdout: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:15:42.583 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2023-07-20T01:15:42.583 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-20T01:15:42.583 INFO:teuthology.orchestra.run.smithi060.stdout: automake-1.16.1-7.el8.noarch 2023-07-20T01:15:42.584 INFO:teuthology.orchestra.run.smithi060.stdout: bison-3.0.4-10.el8.x86_64 2023-07-20T01:15:42.584 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:15:42.584 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:15:42.584 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:15:42.584 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:15:42.585 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:15:42.585 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:15:42.585 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:15:42.585 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:15:42.585 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:15:42.585 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:15:42.585 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:15:42.586 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:15:42.586 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:15:42.586 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:15:42.586 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:15:42.586 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:15:42.586 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:15:42.586 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:15:42.587 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:15:42.587 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:15:42.587 INFO:teuthology.orchestra.run.smithi060.stdout: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:15:42.587 INFO:teuthology.orchestra.run.smithi060.stdout: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:15:42.587 INFO:teuthology.orchestra.run.smithi060.stdout: cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:15:42.587 INFO:teuthology.orchestra.run.smithi060.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-20T01:15:42.587 INFO:teuthology.orchestra.run.smithi060.stdout: flex-2.6.1-9.el8.x86_64 2023-07-20T01:15:42.588 INFO:teuthology.orchestra.run.smithi060.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-20T01:15:42.588 INFO:teuthology.orchestra.run.smithi060.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-20T01:15:42.588 INFO:teuthology.orchestra.run.smithi060.stdout: jq-1.6-3.el8.x86_64 2023-07-20T01:15:42.588 INFO:teuthology.orchestra.run.smithi060.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-20T01:15:42.588 INFO:teuthology.orchestra.run.smithi060.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-20T01:15:42.588 INFO:teuthology.orchestra.run.smithi060.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-20T01:15:42.589 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:15:42.589 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:15:42.589 INFO:teuthology.orchestra.run.smithi060.stdout: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:15:42.589 INFO:teuthology.orchestra.run.smithi060.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-20T01:15:42.589 INFO:teuthology.orchestra.run.smithi060.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-20T01:15:42.589 INFO:teuthology.orchestra.run.smithi060.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-20T01:15:42.589 INFO:teuthology.orchestra.run.smithi060.stdout: libpq-13.5-1.el8.x86_64 2023-07-20T01:15:42.590 INFO:teuthology.orchestra.run.smithi060.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-20T01:15:42.590 INFO:teuthology.orchestra.run.smithi060.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-20T01:15:42.590 INFO:teuthology.orchestra.run.smithi060.stdout: librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:15:42.590 INFO:teuthology.orchestra.run.smithi060.stdout: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:15:42.590 INFO:teuthology.orchestra.run.smithi060.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-20T01:15:42.590 INFO:teuthology.orchestra.run.smithi060.stdout: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:15:42.590 INFO:teuthology.orchestra.run.smithi060.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-20T01:15:42.591 INFO:teuthology.orchestra.run.smithi060.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-20T01:15:42.591 INFO:teuthology.orchestra.run.smithi060.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-20T01:15:42.591 INFO:teuthology.orchestra.run.smithi060.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-20T01:15:42.591 INFO:teuthology.orchestra.run.smithi060.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-20T01:15:42.591 INFO:teuthology.orchestra.run.smithi060.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-20T01:15:42.591 INFO:teuthology.orchestra.run.smithi060.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-20T01:15:42.591 INFO:teuthology.orchestra.run.smithi060.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-20T01:15:42.592 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-20T01:15:42.592 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-20T01:15:42.592 INFO:teuthology.orchestra.run.smithi060.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-20T01:15:42.592 INFO:teuthology.orchestra.run.smithi060.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-20T01:15:42.592 INFO:teuthology.orchestra.run.smithi060.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-20T01:15:42.592 INFO:teuthology.orchestra.run.smithi060.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-20T01:15:42.593 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T01:15:42.593 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T01:15:42.593 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T01:15:42.593 INFO:teuthology.orchestra.run.smithi060.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-20T01:15:42.593 INFO:teuthology.orchestra.run.smithi060.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-20T01:15:42.593 INFO:teuthology.orchestra.run.smithi060.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-20T01:15:42.594 INFO:teuthology.orchestra.run.smithi060.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-20T01:15:42.594 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-20T01:15:42.594 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:15:42.594 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:15:42.594 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:15:42.594 INFO:teuthology.orchestra.run.smithi060.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-20T01:15:42.594 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-20T01:15:42.595 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-20T01:15:42.595 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-20T01:15:42.595 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-20T01:15:42.595 INFO:teuthology.orchestra.run.smithi060.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-20T01:15:42.595 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-20T01:15:42.595 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-20T01:15:42.595 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-20T01:15:42.596 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-20T01:15:42.596 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-20T01:15:42.596 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-20T01:15:42.596 INFO:teuthology.orchestra.run.smithi060.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-20T01:15:42.596 INFO:teuthology.orchestra.run.smithi060.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-20T01:15:42.596 INFO:teuthology.orchestra.run.smithi060.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-20T01:15:42.597 INFO:teuthology.orchestra.run.smithi060.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-20T01:15:42.597 INFO:teuthology.orchestra.run.smithi060.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-20T01:15:42.597 INFO:teuthology.orchestra.run.smithi060.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-20T01:15:42.597 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-20T01:15:42.597 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-20T01:15:42.597 INFO:teuthology.orchestra.run.smithi060.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-20T01:15:42.597 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-20T01:15:42.598 INFO:teuthology.orchestra.run.smithi060.stdout: python3-portend-2.6-1.el8.noarch 2023-07-20T01:15:42.598 INFO:teuthology.orchestra.run.smithi060.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-20T01:15:42.598 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-20T01:15:42.598 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-20T01:15:42.598 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-20T01:15:42.598 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-20T01:15:42.598 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-20T01:15:42.599 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:15:42.599 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:15:42.599 INFO:teuthology.orchestra.run.smithi060.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-20T01:15:42.599 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-20T01:15:42.599 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:15:42.599 INFO:teuthology.orchestra.run.smithi060.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-20T01:15:42.599 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-20T01:15:42.600 INFO:teuthology.orchestra.run.smithi060.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-20T01:15:42.600 INFO:teuthology.orchestra.run.smithi060.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-20T01:15:42.600 INFO:teuthology.orchestra.run.smithi060.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-20T01:15:42.600 INFO:teuthology.orchestra.run.smithi060.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-20T01:15:42.600 INFO:teuthology.orchestra.run.smithi060.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-20T01:15:42.600 INFO:teuthology.orchestra.run.smithi060.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-20T01:15:42.600 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-20T01:15:42.601 INFO:teuthology.orchestra.run.smithi060.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-20T01:15:42.601 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-20T01:15:42.601 INFO:teuthology.orchestra.run.smithi060.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-20T01:15:42.601 INFO:teuthology.orchestra.run.smithi060.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-20T01:15:42.601 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:15:42.601 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:15:42.601 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:15:42.602 INFO:teuthology.orchestra.run.smithi060.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-20T01:15:42.602 INFO:teuthology.orchestra.run.smithi060.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-20T01:15:42.602 INFO:teuthology.orchestra.run.smithi060.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-20T01:15:42.602 INFO:teuthology.orchestra.run.smithi060.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-20T01:15:42.602 INFO:teuthology.orchestra.run.smithi060.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-20T01:15:42.603 INFO:teuthology.orchestra.run.smithi060.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-20T01:15:42.603 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:15:42.603 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2023-07-20T01:15:42.790 DEBUG:teuthology.parallel:result is None 2023-07-20T01:16:33.312 INFO:teuthology.orchestra.run.smithi136.stdout:Extra Packages for Enterprise Linux 57 kB/s | 16 MB 04:38 2023-07-20T01:16:36.811 INFO:teuthology.orchestra.run.smithi136.stdout:lab-extras 504 kB/s | 24 kB 00:00 2023-07-20T01:16:37.796 INFO:teuthology.orchestra.run.smithi136.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 58 MB/s | 53 MB 00:00 2023-07-20T01:16:45.872 INFO:teuthology.orchestra.run.smithi136.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 60 MB/s | 47 MB 00:00 2023-07-20T01:16:56.873 INFO:teuthology.orchestra.run.smithi136.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T01:16:56.874 INFO:teuthology.orchestra.run.smithi136.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-20T01:16:56.889 INFO:teuthology.orchestra.run.smithi136.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-20T01:16:56.889 INFO:teuthology.orchestra.run.smithi136.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-20T01:16:56.890 INFO:teuthology.orchestra.run.smithi136.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-20T01:16:56.891 INFO:teuthology.orchestra.run.smithi136.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-20T01:16:56.894 INFO:teuthology.orchestra.run.smithi136.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-20T01:16:56.897 INFO:teuthology.orchestra.run.smithi136.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-20T01:16:56.898 INFO:teuthology.orchestra.run.smithi136.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-20T01:16:57.027 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2023-07-20T01:16:57.039 INFO:teuthology.orchestra.run.smithi136.stdout:=========================================================================================================================== 2023-07-20T01:16:57.040 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2023-07-20T01:16:57.040 INFO:teuthology.orchestra.run.smithi136.stdout:=========================================================================================================================== 2023-07-20T01:16:57.040 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2023-07-20T01:16:57.040 INFO:teuthology.orchestra.run.smithi136.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-20T01:16:57.040 INFO:teuthology.orchestra.run.smithi136.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-20T01:16:57.040 INFO:teuthology.orchestra.run.smithi136.stdout: ceph x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.4 k 2023-07-20T01:16:57.041 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 5.4 M 2023-07-20T01:16:57.041 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 850 k 2023-07-20T01:16:57.041 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 167 k 2023-07-20T01:16:57.041 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 1.6 M 2023-07-20T01:16:57.041 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 129 k 2023-07-20T01:16:57.041 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 1.7 M 2023-07-20T01:16:57.041 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 7.4 M 2023-07-20T01:16:57.042 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-rook noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 47 k 2023-07-20T01:16:57.042 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-radosgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 14 M 2023-07-20T01:16:57.042 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-test x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 48 M 2023-07-20T01:16:57.042 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-volume noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 258 k 2023-07-20T01:16:57.042 INFO:teuthology.orchestra.run.smithi136.stdout: cephadm noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 165 k 2023-07-20T01:16:57.042 INFO:teuthology.orchestra.run.smithi136.stdout: cephfs-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 260 k 2023-07-20T01:16:57.042 INFO:teuthology.orchestra.run.smithi136.stdout: cephfs-top noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 18 k 2023-07-20T01:16:57.043 INFO:teuthology.orchestra.run.smithi136.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-20T01:16:57.043 INFO:teuthology.orchestra.run.smithi136.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-20T01:16:57.043 INFO:teuthology.orchestra.run.smithi136.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-20T01:16:57.043 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 31 k 2023-07-20T01:16:57.043 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 771 k 2023-07-20T01:16:57.043 INFO:teuthology.orchestra.run.smithi136.stdout: librados-devel x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T01:16:57.043 INFO:teuthology.orchestra.run.smithi136.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-20T01:16:57.043 INFO:teuthology.orchestra.run.smithi136.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-20T01:16:57.044 INFO:teuthology.orchestra.run.smithi136.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-20T01:16:57.044 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-20T01:16:57.044 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-20T01:16:57.044 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-20T01:16:57.044 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cephfs x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 197 k 2023-07-20T01:16:57.044 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 370 k 2023-07-20T01:16:57.044 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 363 k 2023-07-20T01:16:57.045 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 107 k 2023-07-20T01:16:57.045 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-fuse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 94 k 2023-07-20T01:16:57.045 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-mirror x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.5 M 2023-07-20T01:16:57.045 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-nbd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 179 k 2023-07-20T01:16:57.045 INFO:teuthology.orchestra.run.smithi136.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-20T01:16:57.045 INFO:teuthology.orchestra.run.smithi136.stdout:Upgrading: 2023-07-20T01:16:57.045 INFO:teuthology.orchestra.run.smithi136.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-20T01:16:57.046 INFO:teuthology.orchestra.run.smithi136.stdout: librados2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.7 M 2023-07-20T01:16:57.046 INFO:teuthology.orchestra.run.smithi136.stdout: librbd1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 3.6 M 2023-07-20T01:16:57.046 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2023-07-20T01:16:57.046 INFO:teuthology.orchestra.run.smithi136.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-20T01:16:57.046 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 22 M 2023-07-20T01:16:57.046 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-grafana-dashboards noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 23 k 2023-07-20T01:16:57.046 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mds x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 2.3 M 2023-07-20T01:16:57.047 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 241 k 2023-07-20T01:16:57.047 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 4.6 M 2023-07-20T01:16:57.047 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-osd x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 18 M 2023-07-20T01:16:57.047 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-prometheus-alerts noarch 2:18.0.0-5040.g1b0b2eb0.el8 ceph-noarch 15 k 2023-07-20T01:16:57.047 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-selinux x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 24 k 2023-07-20T01:16:57.047 INFO:teuthology.orchestra.run.smithi136.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-20T01:16:57.047 INFO:teuthology.orchestra.run.smithi136.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-20T01:16:57.047 INFO:teuthology.orchestra.run.smithi136.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-20T01:16:57.048 INFO:teuthology.orchestra.run.smithi136.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-20T01:16:57.048 INFO:teuthology.orchestra.run.smithi136.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-20T01:16:57.048 INFO:teuthology.orchestra.run.smithi136.stdout: libcephsqlite x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 139 k 2023-07-20T01:16:57.048 INFO:teuthology.orchestra.run.smithi136.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-20T01:16:57.048 INFO:teuthology.orchestra.run.smithi136.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-20T01:16:57.048 INFO:teuthology.orchestra.run.smithi136.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-20T01:16:57.048 INFO:teuthology.orchestra.run.smithi136.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-20T01:16:57.049 INFO:teuthology.orchestra.run.smithi136.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-20T01:16:57.049 INFO:teuthology.orchestra.run.smithi136.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-20T01:16:57.049 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 472 k 2023-07-20T01:16:57.049 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-20T01:16:57.049 INFO:teuthology.orchestra.run.smithi136.stdout: librgw2 x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 6.8 M 2023-07-20T01:16:57.049 INFO:teuthology.orchestra.run.smithi136.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-20T01:16:57.050 INFO:teuthology.orchestra.run.smithi136.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-20T01:16:57.050 INFO:teuthology.orchestra.run.smithi136.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-20T01:16:57.050 INFO:teuthology.orchestra.run.smithi136.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-20T01:16:57.050 INFO:teuthology.orchestra.run.smithi136.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-20T01:16:57.050 INFO:teuthology.orchestra.run.smithi136.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-20T01:16:57.050 INFO:teuthology.orchestra.run.smithi136.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-20T01:16:57.050 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-20T01:16:57.050 INFO:teuthology.orchestra.run.smithi136.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-20T01:16:57.051 INFO:teuthology.orchestra.run.smithi136.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-20T01:16:57.051 INFO:teuthology.orchestra.run.smithi136.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-20T01:16:57.051 INFO:teuthology.orchestra.run.smithi136.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-20T01:16:57.051 INFO:teuthology.orchestra.run.smithi136.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-20T01:16:57.051 INFO:teuthology.orchestra.run.smithi136.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-20T01:16:57.051 INFO:teuthology.orchestra.run.smithi136.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-20T01:16:57.051 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-20T01:16:57.051 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 45 k 2023-07-20T01:16:57.052 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common x86_64 2:18.0.0-5040.g1b0b2eb0.el8 ceph 122 k 2023-07-20T01:16:57.052 INFO:teuthology.orchestra.run.smithi136.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-20T01:16:57.052 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-20T01:16:57.052 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-20T01:16:57.052 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-20T01:16:57.052 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-20T01:16:57.052 INFO:teuthology.orchestra.run.smithi136.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-20T01:16:57.053 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-20T01:16:57.053 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-20T01:16:57.053 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-20T01:16:57.053 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-20T01:16:57.054 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-20T01:16:57.054 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T01:16:57.054 INFO:teuthology.orchestra.run.smithi136.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-20T01:16:57.054 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-20T01:16:57.054 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-20T01:16:57.054 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-20T01:16:57.054 INFO:teuthology.orchestra.run.smithi136.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-20T01:16:57.055 INFO:teuthology.orchestra.run.smithi136.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-20T01:16:57.055 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-20T01:16:57.055 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-20T01:16:57.055 INFO:teuthology.orchestra.run.smithi136.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-20T01:16:57.056 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-20T01:16:57.056 INFO:teuthology.orchestra.run.smithi136.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-20T01:16:57.056 INFO:teuthology.orchestra.run.smithi136.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-20T01:16:57.056 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-20T01:16:57.056 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-20T01:16:57.056 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-20T01:16:57.056 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-20T01:16:57.057 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-20T01:16:57.057 INFO:teuthology.orchestra.run.smithi136.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-20T01:16:57.057 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-20T01:16:57.057 INFO:teuthology.orchestra.run.smithi136.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-20T01:16:57.057 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-20T01:16:57.057 INFO:teuthology.orchestra.run.smithi136.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-20T01:16:57.057 INFO:teuthology.orchestra.run.smithi136.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-20T01:16:57.057 INFO:teuthology.orchestra.run.smithi136.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-20T01:16:57.058 INFO:teuthology.orchestra.run.smithi136.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-20T01:16:57.058 INFO:teuthology.orchestra.run.smithi136.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-20T01:16:57.058 INFO:teuthology.orchestra.run.smithi136.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-20T01:16:57.058 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-20T01:16:57.058 INFO:teuthology.orchestra.run.smithi136.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-20T01:16:57.058 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-20T01:16:57.058 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-20T01:16:57.058 INFO:teuthology.orchestra.run.smithi136.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-20T01:16:57.059 INFO:teuthology.orchestra.run.smithi136.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-20T01:16:57.059 INFO:teuthology.orchestra.run.smithi136.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-20T01:16:57.059 INFO:teuthology.orchestra.run.smithi136.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-20T01:16:57.059 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-20T01:16:57.059 INFO:teuthology.orchestra.run.smithi136.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-20T01:16:57.059 INFO:teuthology.orchestra.run.smithi136.stdout:Enabling module streams: 2023-07-20T01:16:57.059 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql 10 2023-07-20T01:16:57.060 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-07-20T01:16:57.060 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2023-07-20T01:16:57.060 INFO:teuthology.orchestra.run.smithi136.stdout:=========================================================================================================================== 2023-07-20T01:16:57.060 INFO:teuthology.orchestra.run.smithi136.stdout:Install 128 Packages 2023-07-20T01:16:57.060 INFO:teuthology.orchestra.run.smithi136.stdout:Upgrade 3 Packages 2023-07-20T01:16:57.060 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-07-20T01:16:57.061 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 206 M 2023-07-20T01:16:57.061 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2023-07-20T01:16:58.271 INFO:teuthology.orchestra.run.smithi136.stdout:(1/131): ceph-18.0.0-5040.g1b0b2eb0.el8.x86_64. 17 kB/s | 6.4 kB 00:00 2023-07-20T01:16:59.896 INFO:teuthology.orchestra.run.smithi136.stdout:(2/131): ceph-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 523 kB/s | 850 kB 00:01 2023-07-20T01:17:00.121 INFO:teuthology.orchestra.run.smithi136.stdout:(3/131): ceph-immutable-object-cache-18.0.0-504 743 kB/s | 167 kB 00:00 2023-07-20T01:17:01.721 INFO:teuthology.orchestra.run.smithi136.stdout:(4/131): ceph-mds-18.0.0-5040.g1b0b2eb0.el8.x86 1.4 MB/s | 2.3 MB 00:01 2023-07-20T01:17:02.388 INFO:teuthology.orchestra.run.smithi136.stdout:(5/131): ceph-base-18.0.0-5040.g1b0b2eb0.el8.x8 1.2 MB/s | 5.4 MB 00:04 2023-07-20T01:17:02.422 INFO:teuthology.orchestra.run.smithi136.stdout:(6/131): ceph-mgr-18.0.0-5040.g1b0b2eb0.el8.x86 2.2 MB/s | 1.6 MB 00:00 2023-07-20T01:17:05.188 INFO:teuthology.orchestra.run.smithi136.stdout:(7/131): ceph-mon-18.0.0-5040.g1b0b2eb0.el8.x86 1.6 MB/s | 4.6 MB 00:02 2023-07-20T01:17:13.854 INFO:teuthology.orchestra.run.smithi136.stdout:(8/131): ceph-common-18.0.0-5040.g1b0b2eb0.el8. 1.4 MB/s | 22 MB 00:15 2023-07-20T01:17:14.012 INFO:teuthology.orchestra.run.smithi136.stdout:(9/131): ceph-selinux-18.0.0-5040.g1b0b2eb0.el8 154 kB/s | 24 kB 00:00 2023-07-20T01:17:21.119 INFO:teuthology.orchestra.run.smithi136.stdout:(10/131): ceph-osd-18.0.0-5040.g1b0b2eb0.el8.x8 973 kB/s | 18 MB 00:18 2023-07-20T01:17:21.544 INFO:teuthology.orchestra.run.smithi136.stdout:(11/131): cephfs-mirror-18.0.0-5040.g1b0b2eb0.e 613 kB/s | 260 kB 00:00 2023-07-20T01:17:21.714 INFO:teuthology.orchestra.run.smithi136.stdout:(12/131): libcephfs-devel-18.0.0-5040.g1b0b2eb0 183 kB/s | 31 kB 00:00 2023-07-20T01:17:22.728 INFO:teuthology.orchestra.run.smithi136.stdout:(13/131): libcephfs2-18.0.0-5040.g1b0b2eb0.el8. 761 kB/s | 771 kB 00:01 2023-07-20T01:17:22.987 INFO:teuthology.orchestra.run.smithi136.stdout:(14/131): libcephsqlite-18.0.0-5040.g1b0b2eb0.e 537 kB/s | 139 kB 00:00 2023-07-20T01:17:23.187 INFO:teuthology.orchestra.run.smithi136.stdout:(15/131): librados-devel-18.0.0-5040.g1b0b2eb0. 695 kB/s | 139 kB 00:00 2023-07-20T01:17:23.704 INFO:teuthology.orchestra.run.smithi136.stdout:(16/131): libradosstriper1-18.0.0-5040.g1b0b2eb 914 kB/s | 472 kB 00:00 2023-07-20T01:17:25.445 INFO:teuthology.orchestra.run.smithi136.stdout:(17/131): ceph-radosgw-18.0.0-5040.g1b0b2eb0.el 685 kB/s | 14 MB 00:20 2023-07-20T01:17:25.595 INFO:teuthology.orchestra.run.smithi136.stdout:(18/131): python3-ceph-argparse-18.0.0-5040.g1b 298 kB/s | 45 kB 00:00 2023-07-20T01:17:25.946 INFO:teuthology.orchestra.run.smithi136.stdout:(19/131): python3-ceph-common-18.0.0-5040.g1b0b 350 kB/s | 122 kB 00:00 2023-07-20T01:17:26.446 INFO:teuthology.orchestra.run.smithi136.stdout:(20/131): python3-cephfs-18.0.0-5040.g1b0b2eb0. 394 kB/s | 197 kB 00:00 2023-07-20T01:17:26.904 INFO:teuthology.orchestra.run.smithi136.stdout:(21/131): python3-rados-18.0.0-5040.g1b0b2eb0.e 808 kB/s | 370 kB 00:00 2023-07-20T01:17:27.313 INFO:teuthology.orchestra.run.smithi136.stdout:(22/131): python3-rbd-18.0.0-5040.g1b0b2eb0.el8 889 kB/s | 363 kB 00:00 2023-07-20T01:17:27.672 INFO:teuthology.orchestra.run.smithi136.stdout:(23/131): python3-rgw-18.0.0-5040.g1b0b2eb0.el8 298 kB/s | 107 kB 00:00 2023-07-20T01:17:27.805 INFO:teuthology.orchestra.run.smithi136.stdout:(24/131): rbd-fuse-18.0.0-5040.g1b0b2eb0.el8.x8 702 kB/s | 94 kB 00:00 2023-07-20T01:17:31.147 INFO:teuthology.orchestra.run.smithi136.stdout:(25/131): rbd-mirror-18.0.0-5040.g1b0b2eb0.el8. 1.1 MB/s | 3.5 MB 00:03 2023-07-20T01:17:31.339 INFO:teuthology.orchestra.run.smithi136.stdout:(26/131): rbd-nbd-18.0.0-5040.g1b0b2eb0.el8.x86 934 kB/s | 179 kB 00:00 2023-07-20T01:17:31.414 INFO:teuthology.orchestra.run.smithi136.stdout:(27/131): ceph-grafana-dashboards-18.0.0-5040.g 308 kB/s | 23 kB 00:00 2023-07-20T01:17:31.556 INFO:teuthology.orchestra.run.smithi136.stdout:(28/131): ceph-mgr-cephadm-18.0.0-5040.g1b0b2eb 913 kB/s | 129 kB 00:00 2023-07-20T01:17:32.340 INFO:teuthology.orchestra.run.smithi136.stdout:(29/131): librgw2-18.0.0-5040.g1b0b2eb0.el8.x86 803 kB/s | 6.8 MB 00:08 2023-07-20T01:17:32.981 INFO:teuthology.orchestra.run.smithi136.stdout:(30/131): ceph-mgr-dashboard-18.0.0-5040.g1b0b2 1.2 MB/s | 1.7 MB 00:01 2023-07-20T01:17:33.232 INFO:teuthology.orchestra.run.smithi136.stdout:(31/131): ceph-mgr-modules-core-18.0.0-5040.g1b 964 kB/s | 241 kB 00:00 2023-07-20T01:17:33.324 INFO:teuthology.orchestra.run.smithi136.stdout:(32/131): ceph-mgr-rook-18.0.0-5040.g1b0b2eb0.e 518 kB/s | 47 kB 00:00 2023-07-20T01:17:33.407 INFO:teuthology.orchestra.run.smithi136.stdout:(33/131): ceph-prometheus-alerts-18.0.0-5040.g1 175 kB/s | 15 kB 00:00 2023-07-20T01:17:33.624 INFO:teuthology.orchestra.run.smithi136.stdout:(34/131): ceph-volume-18.0.0-5040.g1b0b2eb0.el8 1.2 MB/s | 258 kB 00:00 2023-07-20T01:17:33.783 INFO:teuthology.orchestra.run.smithi136.stdout:(35/131): cephadm-18.0.0-5040.g1b0b2eb0.el8.noa 1.0 MB/s | 165 kB 00:00 2023-07-20T01:17:33.908 INFO:teuthology.orchestra.run.smithi136.stdout:(36/131): cephfs-top-18.0.0-5040.g1b0b2eb0.el8. 144 kB/s | 18 kB 00:00 2023-07-20T01:17:34.608 INFO:teuthology.orchestra.run.smithi136.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 437 kB/s | 306 kB 00:00 2023-07-20T01:17:35.150 INFO:teuthology.orchestra.run.smithi136.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 109 kB/s | 59 kB 00:00 2023-07-20T01:17:35.542 INFO:teuthology.orchestra.run.smithi136.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 192 kB/s | 75 kB 00:00 2023-07-20T01:17:36.661 INFO:teuthology.orchestra.run.smithi136.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 425 kB/s | 476 kB 00:01 2023-07-20T01:17:36.809 INFO:teuthology.orchestra.run.smithi136.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 295 kB/s | 44 kB 00:00 2023-07-20T01:17:37.168 INFO:teuthology.orchestra.run.smithi136.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 517 kB/s | 185 kB 00:00 2023-07-20T01:17:37.301 INFO:teuthology.orchestra.run.smithi136.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 248 kB/s | 33 kB 00:00 2023-07-20T01:17:37.410 INFO:teuthology.orchestra.run.smithi136.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 150 kB/s | 16 kB 00:00 2023-07-20T01:17:37.868 INFO:teuthology.orchestra.run.smithi136.stdout:(45/131): python3-cheroot-8.5.2-1.el8.noarch.rp 377 kB/s | 173 kB 00:00 2023-07-20T01:17:40.200 INFO:teuthology.orchestra.run.smithi136.stdout:(46/131): python3-cherrypy-18.4.0-1.el8.noarch. 165 kB/s | 384 kB 00:02 2023-07-20T01:17:40.377 INFO:teuthology.orchestra.run.smithi136.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 460 kB/s | 81 kB 00:00 2023-07-20T01:17:40.452 INFO:teuthology.orchestra.run.smithi136.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 148 kB/s | 11 kB 00:00 2023-07-20T01:17:40.553 INFO:teuthology.orchestra.run.smithi136.stdout:(49/131): python3-jaraco-functools-2.0-4.el8.no 175 kB/s | 18 kB 00:00 2023-07-20T01:17:41.545 INFO:teuthology.orchestra.run.smithi136.stdout:(50/131): python3-kubernetes-11.0.0-6.el8.noarc 1.2 MB/s | 1.2 MB 00:00 2023-07-20T01:17:41.636 INFO:teuthology.orchestra.run.smithi136.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 532 kB/s | 49 kB 00:00 2023-07-20T01:17:41.737 INFO:teuthology.orchestra.run.smithi136.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 588 kB/s | 59 kB 00:00 2023-07-20T01:17:41.845 INFO:teuthology.orchestra.run.smithi136.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 557 kB/s | 60 kB 00:00 2023-07-20T01:17:42.054 INFO:teuthology.orchestra.run.smithi136.stdout:(54/131): python3-pecan-1.3.2-9.el8.noarch.rpm 1.3 MB/s | 283 kB 00:00 2023-07-20T01:17:42.113 INFO:teuthology.orchestra.run.smithi136.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 267 kB/s | 16 kB 00:00 2023-07-20T01:17:42.196 INFO:teuthology.orchestra.run.smithi136.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 405 kB/s | 34 kB 00:00 2023-07-20T01:17:42.338 INFO:teuthology.orchestra.run.smithi136.stdout:(57/131): python3-routes-2.4.1-12.el8.noarch.rp 1.4 MB/s | 196 kB 00:00 2023-07-20T01:17:42.430 INFO:teuthology.orchestra.run.smithi136.stdout:(58/131): ceph-mgr-diskprediction-local-18.0.0- 752 kB/s | 7.4 MB 00:10 2023-07-20T01:17:42.447 INFO:teuthology.orchestra.run.smithi136.stdout:(59/131): python3-rsa-4.8-1.el8.noarch.rpm 566 kB/s | 61 kB 00:00 2023-07-20T01:17:42.573 INFO:teuthology.orchestra.run.smithi136.stdout:(60/131): python3-singledispatch-3.4.0.3-18.el8 191 kB/s | 24 kB 00:00 2023-07-20T01:17:42.656 INFO:teuthology.orchestra.run.smithi136.stdout:(61/131): python3-tempora-1.14.1-5.el8.noarch.r 351 kB/s | 29 kB 00:00 2023-07-20T01:17:42.740 INFO:teuthology.orchestra.run.smithi136.stdout:(62/131): python3-trustme-0.6.0-4.el8.noarch.rp 331 kB/s | 27 kB 00:00 2023-07-20T01:17:43.082 INFO:teuthology.orchestra.run.smithi136.stdout:(63/131): python3-waitress-1.4.3-1.el8.noarch.r 747 kB/s | 255 kB 00:00 2023-07-20T01:17:43.682 INFO:teuthology.orchestra.run.smithi136.stdout:(64/131): python3-simplegeneric-0.8.1-17.el8.no 15 kB/s | 19 kB 00:01 2023-07-20T01:17:43.991 INFO:teuthology.orchestra.run.smithi136.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 199 kB/s | 61 kB 00:00 2023-07-20T01:17:44.282 INFO:teuthology.orchestra.run.smithi136.stdout:(66/131): python3-webtest-2.0.33-1.el8.noarch.r 293 kB/s | 85 kB 00:00 2023-07-20T01:17:44.416 INFO:teuthology.orchestra.run.smithi136.stdout:(67/131): python3-zc-lockfile-2.0-2.el8.noarch. 172 kB/s | 23 kB 00:00 2023-07-20T01:17:44.816 INFO:teuthology.orchestra.run.smithi136.stdout:(68/131): python3-webob-1.8.5-1.el8.1.noarch.rp 145 kB/s | 251 kB 00:01 2023-07-20T01:17:44.841 INFO:teuthology.orchestra.run.smithi136.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-07-20T01:17:44.925 INFO:teuthology.orchestra.run.smithi136.stdout:(70/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 562 kB/s | 47 kB 00:00 2023-07-20T01:17:44.967 INFO:teuthology.orchestra.run.smithi136.stdout:(71/131): libselinux-devel-2.9-5.el8.x86_64.rpm 4.7 MB/s | 200 kB 00:00 2023-07-20T01:17:45.009 INFO:teuthology.orchestra.run.smithi136.stdout:(72/131): python3-jwt-1.6.1-2.el8.noarch.rpm 1.0 MB/s | 43 kB 00:00 2023-07-20T01:17:45.051 INFO:teuthology.orchestra.run.smithi136.stdout:(73/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 5.2 MB/s | 220 kB 00:00 2023-07-20T01:17:45.093 INFO:teuthology.orchestra.run.smithi136.stdout:(74/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 1.1 MB/s | 48 kB 00:00 2023-07-20T01:17:45.137 INFO:teuthology.orchestra.run.smithi136.stdout:(75/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.9 MB/s | 87 kB 00:00 2023-07-20T01:17:45.179 INFO:teuthology.orchestra.run.smithi136.stdout:(76/131): python3-requests-oauthlib-1.0.0-1.el8 1.0 MB/s | 43 kB 00:00 2023-07-20T01:17:45.496 INFO:teuthology.orchestra.run.smithi136.stdout:(77/131): python3-pycparser-2.14-14.el8.noarch. 344 kB/s | 109 kB 00:00 2023-07-20T01:17:45.538 INFO:teuthology.orchestra.run.smithi136.stdout:(78/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 5.4 MB/s | 229 kB 00:00 2023-07-20T01:17:45.588 INFO:teuthology.orchestra.run.smithi136.stdout:(79/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 11 MB/s | 560 kB 00:00 2023-07-20T01:17:45.622 INFO:teuthology.orchestra.run.smithi136.stdout:(80/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 562 kB/s | 19 kB 00:00 2023-07-20T01:17:45.664 INFO:teuthology.orchestra.run.smithi136.stdout:(81/131): python3-oauthlib-2.1.0-1.el8.noarch.r 3.6 MB/s | 155 kB 00:00 2023-07-20T01:17:45.745 INFO:teuthology.orchestra.run.smithi136.stdout:(82/131): python3-cryptography-3.2.1-5.el8.x86_ 6.7 MB/s | 559 kB 00:00 2023-07-20T01:17:45.781 INFO:teuthology.orchestra.run.smithi136.stdout:(83/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 504 kB/s | 18 kB 00:00 2023-07-20T01:17:45.832 INFO:teuthology.orchestra.run.smithi136.stdout:(84/131): pcre2-devel-10.32-2.el8.x86_64.rpm 12 MB/s | 605 kB 00:00 2023-07-20T01:17:45.874 INFO:teuthology.orchestra.run.smithi136.stdout:(85/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 5.6 MB/s | 238 kB 00:00 2023-07-20T01:17:45.957 INFO:teuthology.orchestra.run.smithi136.stdout:(86/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 28 MB/s | 2.3 MB 00:00 2023-07-20T01:17:45.999 INFO:teuthology.orchestra.run.smithi136.stdout:(87/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 1.7 MB/s | 72 kB 00:00 2023-07-20T01:17:46.041 INFO:teuthology.orchestra.run.smithi136.stdout:(88/131): fuse-2.9.7-15.el8.x86_64.rpm 1.9 MB/s | 83 kB 00:00 2023-07-20T01:17:46.083 INFO:teuthology.orchestra.run.smithi136.stdout:(89/131): elfutils-libelf-devel-0.186-1.el8.x86 1.4 MB/s | 60 kB 00:00 2023-07-20T01:17:46.150 INFO:teuthology.orchestra.run.smithi136.stdout:(90/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 1.4 MB/s | 98 kB 00:00 2023-07-20T01:17:46.192 INFO:teuthology.orchestra.run.smithi136.stdout:(91/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 1.4 MB/s | 58 kB 00:00 2023-07-20T01:17:46.226 INFO:teuthology.orchestra.run.smithi136.stdout:(92/131): libcom_err-devel-1.45.6-4.el8.x86_64. 1.1 MB/s | 39 kB 00:00 2023-07-20T01:17:46.276 INFO:teuthology.orchestra.run.smithi136.stdout:(93/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 13 MB/s | 644 kB 00:00 2023-07-20T01:17:46.318 INFO:teuthology.orchestra.run.smithi136.stdout:(94/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 4.0 MB/s | 171 kB 00:00 2023-07-20T01:17:46.360 INFO:teuthology.orchestra.run.smithi136.stdout:(95/131): NetworkManager-initscripts-updown-1.3 3.2 MB/s | 138 kB 00:00 2023-07-20T01:17:46.477 INFO:teuthology.orchestra.run.smithi136.stdout:(96/131): thrift-0.13.0-2.el8.x86_64.rpm 862 kB/s | 1.7 MB 00:02 2023-07-20T01:17:46.552 INFO:teuthology.orchestra.run.smithi136.stdout:(97/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.4 MB/s | 110 kB 00:00 2023-07-20T01:17:46.594 INFO:teuthology.orchestra.run.smithi136.stdout:(98/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.3 MB/s | 54 kB 00:00 2023-07-20T01:17:46.636 INFO:teuthology.orchestra.run.smithi136.stdout:(99/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.1 MB/s | 259 kB 00:00 2023-07-20T01:17:46.770 INFO:teuthology.orchestra.run.smithi136.stdout:(100/131): python3-scipy-1.0.0-21.module+el8.5. 33 MB/s | 14 MB 00:00 2023-07-20T01:17:46.804 INFO:teuthology.orchestra.run.smithi136.stdout:(101/131): socat-1.7.4.1-1.el8.x86_64.rpm 1.9 MB/s | 323 kB 00:00 2023-07-20T01:17:46.829 INFO:teuthology.orchestra.run.smithi136.stdout:(102/131): python3-markupsafe-0.23-19.el8.x86_6 661 kB/s | 39 kB 00:00 2023-07-20T01:17:46.863 INFO:teuthology.orchestra.run.smithi136.stdout:(103/131): bison-3.0.4-10.el8.x86_64.rpm 11 MB/s | 688 kB 00:00 2023-07-20T01:17:46.923 INFO:teuthology.orchestra.run.smithi136.stdout:(104/131): python3-jinja2-2.10.1-3.el8.noarch.r 5.6 MB/s | 538 kB 00:00 2023-07-20T01:17:46.955 INFO:teuthology.orchestra.run.smithi136.stdout:(105/131): python3-mako-1.0.6-13.el8.noarch.rpm 1.7 MB/s | 157 kB 00:00 2023-07-20T01:17:46.980 INFO:teuthology.orchestra.run.smithi136.stdout:(106/131): oniguruma-6.8.2-2.el8.x86_64.rpm 3.2 MB/s | 187 kB 00:00 2023-07-20T01:17:47.014 INFO:teuthology.orchestra.run.smithi136.stdout:(107/131): automake-1.16.1-7.el8.noarch.rpm 12 MB/s | 713 kB 00:00 2023-07-20T01:17:47.056 INFO:teuthology.orchestra.run.smithi136.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 639 kB/s | 27 kB 00:00 2023-07-20T01:17:47.098 INFO:teuthology.orchestra.run.smithi136.stdout:(109/131): flex-2.6.1-9.el8.x86_64.rpm 7.5 MB/s | 320 kB 00:00 2023-07-20T01:17:47.131 INFO:teuthology.orchestra.run.smithi136.stdout:(110/131): python3-prettytable-0.7.2-14.el8.noa 1.3 MB/s | 44 kB 00:00 2023-07-20T01:17:47.198 INFO:teuthology.orchestra.run.smithi136.stdout:(111/131): python3-babel-2.5.1-7.el8.noarch.rpm 22 MB/s | 4.8 MB 00:00 2023-07-20T01:17:47.240 INFO:teuthology.orchestra.run.smithi136.stdout:(112/131): python3-werkzeug-0.12.2-4.el8.noarch 11 MB/s | 457 kB 00:00 2023-07-20T01:17:47.310 INFO:teuthology.orchestra.run.smithi136.stdout:(113/131): python3-numpy-f2py-1.14.3-10.el8.x86 3.2 MB/s | 225 kB 00:00 2023-07-20T01:17:47.374 INFO:teuthology.orchestra.run.smithi136.stdout:(114/131): python3-numpy-1.14.3-10.el8.x86_64.r 15 MB/s | 3.7 MB 00:00 2023-07-20T01:17:47.391 INFO:teuthology.orchestra.run.smithi136.stdout:(115/131): uuid-1.6.2-43.el8.x86_64.rpm 792 kB/s | 64 kB 00:00 2023-07-20T01:17:47.416 INFO:teuthology.orchestra.run.smithi136.stdout:(116/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 3.0 MB/s | 126 kB 00:00 2023-07-20T01:17:47.433 INFO:teuthology.orchestra.run.smithi136.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 478 kB/s | 20 kB 00:00 2023-07-20T01:17:47.459 INFO:teuthology.orchestra.run.smithi136.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 2.4 MB/s | 103 kB 00:00 2023-07-20T01:17:47.492 INFO:teuthology.orchestra.run.smithi136.stdout:(119/131): libtool-2.4.6-25.el8.x86_64.rpm 12 MB/s | 709 kB 00:00 2023-07-20T01:17:47.534 INFO:teuthology.orchestra.run.smithi136.stdout:(120/131): libpq-13.5-1.el8.x86_64.rpm 4.6 MB/s | 198 kB 00:00 2023-07-20T01:17:47.634 INFO:teuthology.orchestra.run.smithi136.stdout:(121/131): openblas-threads-0.3.15-3.el8.x86_64 28 MB/s | 4.9 MB 00:00 2023-07-20T01:17:47.705 INFO:teuthology.orchestra.run.smithi136.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 2.8 MB/s | 202 kB 00:00 2023-07-20T01:17:47.785 INFO:teuthology.orchestra.run.smithi136.stdout:(123/131): openblas-0.3.15-3.el8.x86_64.rpm 19 MB/s | 4.7 MB 00:00 2023-07-20T01:17:47.810 INFO:teuthology.orchestra.run.smithi136.stdout:(124/131): librdkafka-0.11.4-3.el8.x86_64.rpm 3.3 MB/s | 354 kB 00:00 2023-07-20T01:17:47.869 INFO:teuthology.orchestra.run.smithi136.stdout:(125/131): postgresql-contrib-10.21-2.module+el 13 MB/s | 808 kB 00:00 2023-07-20T01:17:47.961 INFO:teuthology.orchestra.run.smithi136.stdout:(126/131): postgresql-server-10.21-2.module+el8 29 MB/s | 5.1 MB 00:00 2023-07-20T01:17:48.003 INFO:teuthology.orchestra.run.smithi136.stdout:(127/131): postgresql-10.21-2.module+el8.6.0+15 11 MB/s | 1.5 MB 00:00 2023-07-20T01:17:51.086 INFO:teuthology.orchestra.run.smithi136.stdout:(128/131): librbd1-18.0.0-5040.g1b0b2eb0.el8.x8 1.2 MB/s | 3.6 MB 00:03 2023-07-20T01:17:51.136 INFO:teuthology.orchestra.run.smithi136.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 16 MB/s | 798 kB 00:00 2023-07-20T01:17:52.620 INFO:teuthology.orchestra.run.smithi136.stdout:(130/131): librados2-18.0.0-5040.g1b0b2eb0.el8. 809 kB/s | 3.7 MB 00:04 2023-07-20T01:18:04.851 INFO:teuthology.orchestra.run.smithi136.stdout:(131/131): ceph-test-18.0.0-5040.g1b0b2eb0.el8. 966 kB/s | 48 MB 00:50 2023-07-20T01:18:04.854 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2023-07-20T01:18:04.854 INFO:teuthology.orchestra.run.smithi136.stdout:Total 3.0 MB/s | 206 MB 01:07 2023-07-20T01:18:05.654 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2023-07-20T01:18:05.773 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2023-07-20T01:18:05.773 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2023-07-20T01:18:06.878 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2023-07-20T01:18:06.882 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2023-07-20T01:18:11.859 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2023-07-20T01:18:12.046 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-20T01:18:12.254 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-20T01:18:12.431 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-20T01:18:12.577 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-20T01:18:12.798 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-20T01:18:13.245 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-20T01:18:13.490 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 5/134 2023-07-20T01:18:13.869 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 5/134 2023-07-20T01:18:14.081 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T01:18:14.192 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T01:18:14.373 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T01:18:14.506 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T01:18:14.669 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T01:18:14.836 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-07-20T01:18:15.084 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-20T01:18:15.317 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-20T01:18:15.721 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-20T01:18:15.943 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-20T01:18:16.114 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T01:18:16.207 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 14/134 2023-07-20T01:18:16.347 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T01:18:16.479 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 15/134 2023-07-20T01:18:16.803 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-07-20T01:18:17.011 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-07-20T01:18:17.189 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-07-20T01:18:17.355 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-20T01:18:17.484 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-20T01:18:17.672 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-20T01:18:18.284 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-20T01:18:18.480 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-20T01:18:19.092 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-20T01:18:19.330 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-07-20T01:18:20.010 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-07-20T01:18:20.261 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-07-20T01:18:20.494 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-07-20T01:18:20.723 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-20T01:18:20.944 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 26/134 2023-07-20T01:18:21.091 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 27/134 2023-07-20T01:18:21.341 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 28/134 2023-07-20T01:18:21.546 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-20T01:18:23.302 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-20T01:18:24.344 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-07-20T01:18:24.668 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-07-20T01:18:24.990 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-07-20T01:18:25.257 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-07-20T01:18:25.473 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-20T01:18:25.705 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-20T01:18:25.900 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 36/134 2023-07-20T01:18:26.076 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 37/134 2023-07-20T01:18:26.235 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-07-20T01:18:26.429 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-07-20T01:18:26.611 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-07-20T01:18:26.678 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-07-20T01:18:26.942 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-07-20T01:18:27.198 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-07-20T01:18:27.353 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-20T01:18:27.473 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-20T01:18:27.707 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-07-20T01:18:27.873 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-07-20T01:18:28.093 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-07-20T01:18:28.271 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-07-20T01:18:28.497 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-07-20T01:18:28.648 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-07-20T01:18:28.869 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-07-20T01:18:29.101 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-07-20T01:18:29.510 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-07-20T01:18:29.805 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-07-20T01:18:30.064 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-07-20T01:18:30.290 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-07-20T01:18:30.557 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-07-20T01:18:30.885 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-07-20T01:18:31.152 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-07-20T01:18:31.381 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-07-20T01:18:31.649 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-07-20T01:18:31.823 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-07-20T01:18:32.048 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-07-20T01:18:32.363 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-07-20T01:18:32.623 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-07-20T01:18:32.915 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-07-20T01:18:33.194 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-07-20T01:18:33.376 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-20T01:18:33.596 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-20T01:18:33.868 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-20T01:18:34.117 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-20T01:18:34.289 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-20T01:18:34.504 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-20T01:18:34.727 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-20T01:18:34.980 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-20T01:18:35.222 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-20T01:18:35.404 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-20T01:18:35.599 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-20T01:18:35.835 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-20T01:18:36.130 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-20T01:18:36.324 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-20T01:18:36.599 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-20T01:18:36.875 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-20T01:18:37.125 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-20T01:18:37.394 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-20T01:18:38.520 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-20T01:18:38.829 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-20T01:18:39.075 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-20T01:18:39.460 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-20T01:18:39.739 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-20T01:18:39.944 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-20T01:18:40.771 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-20T01:18:40.950 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T01:18:41.021 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 92/134 2023-07-20T01:18:41.217 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 93/134 2023-07-20T01:18:44.829 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T01:18:45.058 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T01:18:45.663 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 94/134 2023-07-20T01:18:45.976 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T01:18:46.056 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 95/134 2023-07-20T01:18:46.224 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T01:19:05.133 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 96/134 2023-07-20T01:19:05.133 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /sys 2023-07-20T01:19:05.133 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /proc 2023-07-20T01:19:05.133 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /dev 2023-07-20T01:19:05.133 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /run 2023-07-20T01:19:05.133 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /mnt 2023-07-20T01:19:05.134 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /var/tmp 2023-07-20T01:19:05.134 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /home 2023-07-20T01:19:05.134 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /tmp 2023-07-20T01:19:05.134 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /dev 2023-07-20T01:19:05.134 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-07-20T01:19:05.520 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T01:19:07.107 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 97/134 2023-07-20T01:19:07.304 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T01:19:07.523 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 98/134 2023-07-20T01:19:07.744 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T01:19:08.197 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 99/134 2023-07-20T01:19:08.344 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T01:19:08.360 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 100/134 2023-07-20T01:19:10.005 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T01:19:10.239 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 101/134 2023-07-20T01:19:10.358 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 102/134 2023-07-20T01:19:10.599 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 103/134 2023-07-20T01:19:10.785 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T01:19:10.999 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 104/134 2023-07-20T01:19:11.203 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T01:19:12.162 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 105/134 2023-07-20T01:19:12.342 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T01:19:12.432 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 106/134 2023-07-20T01:19:12.790 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 107/134 2023-07-20T01:19:13.016 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T01:19:13.051 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 108/134 2023-07-20T01:19:13.250 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T01:19:13.261 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 109/134 2023-07-20T01:19:14.575 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 110/134 2023-07-20T01:19:14.825 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T01:19:14.858 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 111/134 2023-07-20T01:19:15.000 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T01:19:15.042 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 112/134 2023-07-20T01:19:15.225 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T01:19:15.578 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 113/134 2023-07-20T01:19:15.725 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T01:19:20.281 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 114/134 2023-07-20T01:19:21.204 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 115/134 2023-07-20T01:19:21.547 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-07-20T01:19:21.726 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 117/134 2023-07-20T01:19:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-07-20T01:19:22.207 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-07-20T01:19:22.408 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-07-20T01:19:22.686 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-07-20T01:19:22.878 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 121/134 2023-07-20T01:19:23.074 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-20T01:19:25.281 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T01:19:25.581 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T01:19:25.600 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-20T01:19:25.864 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 124/134 2023-07-20T01:19:26.089 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 125/134 2023-07-20T01:19:26.445 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 126/134 2023-07-20T01:19:26.810 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-20T01:19:27.047 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-07-20T01:19:27.132 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-07-20T01:19:27.331 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-07-20T01:19:27.343 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-07-20T01:19:27.531 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-20T01:19:27.538 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-20T01:19:27.674 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-07-20T01:19:27.675 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T01:19:27.900 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T01:19:27.901 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T01:19:28.131 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-20T01:19:28.354 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T01:19:28.837 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T01:19:28.838 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 1/134 2023-07-20T01:19:28.838 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2/134 2023-07-20T01:19:28.838 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 3/134 2023-07-20T01:19:28.838 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 4/134 2023-07-20T01:19:28.838 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2 5/134 2023-07-20T01:19:28.838 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 6/134 2023-07-20T01:19:28.839 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 7/134 2023-07-20T01:19:28.839 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 8/134 2023-07-20T01:19:28.839 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 9/134 2023-07-20T01:19:28.839 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 10/134 2023-07-20T01:19:28.839 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 11/134 2023-07-20T01:19:28.839 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 12/134 2023-07-20T01:19:28.839 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 13/134 2023-07-20T01:19:28.840 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_ 14/134 2023-07-20T01:19:28.840 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 15/134 2023-07-20T01:19:28.840 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 16/134 2023-07-20T01:19:28.840 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 17/134 2023-07-20T01:19:28.840 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86 18/134 2023-07-20T01:19:28.840 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 19/134 2023-07-20T01:19:28.840 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el 20/134 2023-07-20T01:19:28.840 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8. 21/134 2023-07-20T01:19:28.841 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_6 22/134 2023-07-20T01:19:28.841 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 23/134 2023-07-20T01:19:28.841 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 24/134 2023-07-20T01:19:28.841 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 25/134 2023-07-20T01:19:28.841 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 26/134 2023-07-20T01:19:28.841 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 27/134 2023-07-20T01:19:28.841 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 28/134 2023-07-20T01:19:28.842 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0. 29/134 2023-07-20T01:19:28.842 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noa 30/134 2023-07-20T01:19:28.842 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.n 31/134 2023-07-20T01:19:28.842 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0 32/134 2023-07-20T01:19:28.842 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el 33/134 2023-07-20T01:19:28.842 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 34/134 2023-07-20T01:19:28.842 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.e 35/134 2023-07-20T01:19:28.842 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 36/134 2023-07-20T01:19:28.843 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 37/134 2023-07-20T01:19:28.843 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 38/134 2023-07-20T01:19:28.843 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-20T01:19:28.843 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-20T01:19:28.843 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-20T01:19:28.843 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-20T01:19:28.843 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-20T01:19:28.843 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-20T01:19:28.844 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-20T01:19:28.844 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-20T01:19:28.844 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-20T01:19:28.844 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-20T01:19:28.844 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-20T01:19:28.844 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-20T01:19:28.844 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-20T01:19:28.845 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-20T01:19:28.845 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-20T01:19:28.845 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-20T01:19:28.845 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-20T01:19:28.845 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-20T01:19:28.845 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-20T01:19:28.845 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-20T01:19:28.845 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-20T01:19:28.846 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-20T01:19:28.846 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-20T01:19:28.846 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-20T01:19:28.846 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-20T01:19:28.846 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-20T01:19:28.846 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-20T01:19:28.846 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-20T01:19:28.846 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-20T01:19:28.847 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-20T01:19:28.847 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-20T01:19:28.847 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-20T01:19:28.847 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-20T01:19:28.847 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-07-20T01:19:28.847 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-07-20T01:19:28.847 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-07-20T01:19:28.848 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-07-20T01:19:28.848 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-07-20T01:19:28.848 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-07-20T01:19:28.848 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-07-20T01:19:28.848 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-07-20T01:19:28.848 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-07-20T01:19:28.848 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-07-20T01:19:28.848 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-07-20T01:19:28.849 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-07-20T01:19:28.849 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-07-20T01:19:28.849 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-07-20T01:19:28.849 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-07-20T01:19:28.849 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-07-20T01:19:28.849 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-07-20T01:19:28.849 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-07-20T01:19:28.849 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-07-20T01:19:28.850 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-07-20T01:19:28.850 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-07-20T01:19:28.850 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-07-20T01:19:28.850 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-07-20T01:19:28.850 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-07-20T01:19:28.850 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-07-20T01:19:28.850 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-07-20T01:19:28.850 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-07-20T01:19:28.851 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-07-20T01:19:28.851 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-07-20T01:19:28.851 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-07-20T01:19:28.851 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-07-20T01:19:28.851 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-07-20T01:19:28.851 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-07-20T01:19:28.851 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-07-20T01:19:28.852 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-07-20T01:19:28.852 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-07-20T01:19:28.852 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-07-20T01:19:28.852 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-07-20T01:19:28.852 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-07-20T01:19:28.852 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-07-20T01:19:28.852 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-07-20T01:19:28.853 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-07-20T01:19:28.853 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-07-20T01:19:28.853 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-07-20T01:19:28.853 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-07-20T01:19:28.853 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-07-20T01:19:28.853 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-07-20T01:19:28.853 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-07-20T01:19:28.853 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-07-20T01:19:28.854 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-07-20T01:19:28.854 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-07-20T01:19:28.854 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-07-20T01:19:28.854 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-07-20T01:19:28.854 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-07-20T01:19:28.854 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-07-20T01:19:28.854 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-07-20T01:19:28.854 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-07-20T01:19:28.855 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 129/134 2023-07-20T01:19:28.855 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-20T01:19:28.855 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 131/134 2023-07-20T01:19:28.855 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-20T01:19:28.855 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-20T01:19:31.357 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-20T01:19:31.357 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:02:46 ago on Thu 20 Jul 2023 01:16:45 AM UTC. 2023-07-20T01:19:33.924 INFO:teuthology.orchestra.run.smithi136.stdout:Installed products updated. 2023-07-20T01:19:34.136 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-07-20T01:19:34.136 INFO:teuthology.orchestra.run.smithi136.stdout:Upgraded: 2023-07-20T01:19:34.136 INFO:teuthology.orchestra.run.smithi136.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-20T01:19:34.136 INFO:teuthology.orchestra.run.smithi136.stdout: librados2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:19:34.137 INFO:teuthology.orchestra.run.smithi136.stdout: librbd1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:19:34.137 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2023-07-20T01:19:34.137 INFO:teuthology.orchestra.run.smithi136.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-20T01:19:34.137 INFO:teuthology.orchestra.run.smithi136.stdout: automake-1.16.1-7.el8.noarch 2023-07-20T01:19:34.137 INFO:teuthology.orchestra.run.smithi136.stdout: bison-3.0.4-10.el8.x86_64 2023-07-20T01:19:34.137 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:19:34.137 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:19:34.138 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:19:34.138 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:19:34.138 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-grafana-dashboards-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:19:34.138 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-immutable-object-cache-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:19:34.138 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mds-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:19:34.138 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:19:34.138 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:19:34.138 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:19:34.139 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-diskprediction-local-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:19:34.139 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:19:34.139 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-rook-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:19:34.139 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:19:34.139 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-osd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:19:34.139 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-prometheus-alerts-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:19:34.139 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-radosgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:19:34.139 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-selinux-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:19:34.140 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-test-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:19:34.140 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-volume-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:19:34.140 INFO:teuthology.orchestra.run.smithi136.stdout: cephadm-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:19:34.140 INFO:teuthology.orchestra.run.smithi136.stdout: cephfs-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:19:34.140 INFO:teuthology.orchestra.run.smithi136.stdout: cephfs-top-2:18.0.0-5040.g1b0b2eb0.el8.noarch 2023-07-20T01:19:34.140 INFO:teuthology.orchestra.run.smithi136.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-20T01:19:34.140 INFO:teuthology.orchestra.run.smithi136.stdout: flex-2.6.1-9.el8.x86_64 2023-07-20T01:19:34.141 INFO:teuthology.orchestra.run.smithi136.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-20T01:19:34.141 INFO:teuthology.orchestra.run.smithi136.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-20T01:19:34.141 INFO:teuthology.orchestra.run.smithi136.stdout: jq-1.6-3.el8.x86_64 2023-07-20T01:19:34.141 INFO:teuthology.orchestra.run.smithi136.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-20T01:19:34.141 INFO:teuthology.orchestra.run.smithi136.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-20T01:19:34.141 INFO:teuthology.orchestra.run.smithi136.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-20T01:19:34.141 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:19:34.141 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:19:34.142 INFO:teuthology.orchestra.run.smithi136.stdout: libcephsqlite-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:19:34.142 INFO:teuthology.orchestra.run.smithi136.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-20T01:19:34.142 INFO:teuthology.orchestra.run.smithi136.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-20T01:19:34.142 INFO:teuthology.orchestra.run.smithi136.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-20T01:19:34.142 INFO:teuthology.orchestra.run.smithi136.stdout: libpq-13.5-1.el8.x86_64 2023-07-20T01:19:34.142 INFO:teuthology.orchestra.run.smithi136.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-20T01:19:34.142 INFO:teuthology.orchestra.run.smithi136.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-20T01:19:34.142 INFO:teuthology.orchestra.run.smithi136.stdout: librados-devel-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:19:34.143 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:19:34.143 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-20T01:19:34.143 INFO:teuthology.orchestra.run.smithi136.stdout: librgw2-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:19:34.143 INFO:teuthology.orchestra.run.smithi136.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-20T01:19:34.143 INFO:teuthology.orchestra.run.smithi136.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-20T01:19:34.143 INFO:teuthology.orchestra.run.smithi136.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-20T01:19:34.143 INFO:teuthology.orchestra.run.smithi136.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-20T01:19:34.143 INFO:teuthology.orchestra.run.smithi136.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-20T01:19:34.144 INFO:teuthology.orchestra.run.smithi136.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-20T01:19:34.144 INFO:teuthology.orchestra.run.smithi136.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-20T01:19:34.144 INFO:teuthology.orchestra.run.smithi136.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-20T01:19:34.144 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-20T01:19:34.144 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-20T01:19:34.144 INFO:teuthology.orchestra.run.smithi136.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-20T01:19:34.144 INFO:teuthology.orchestra.run.smithi136.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-20T01:19:34.145 INFO:teuthology.orchestra.run.smithi136.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-20T01:19:34.145 INFO:teuthology.orchestra.run.smithi136.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-20T01:19:34.145 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T01:19:34.145 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T01:19:34.145 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-20T01:19:34.145 INFO:teuthology.orchestra.run.smithi136.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-20T01:19:34.145 INFO:teuthology.orchestra.run.smithi136.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-20T01:19:34.145 INFO:teuthology.orchestra.run.smithi136.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-20T01:19:34.146 INFO:teuthology.orchestra.run.smithi136.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-20T01:19:34.146 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-20T01:19:34.146 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:19:34.146 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:19:34.146 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cephfs-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:19:34.146 INFO:teuthology.orchestra.run.smithi136.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-20T01:19:34.146 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-20T01:19:34.146 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-20T01:19:34.147 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-20T01:19:34.147 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-20T01:19:34.147 INFO:teuthology.orchestra.run.smithi136.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-20T01:19:34.147 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-20T01:19:34.147 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-20T01:19:34.147 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-20T01:19:34.147 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-20T01:19:34.147 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-20T01:19:34.148 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-20T01:19:34.148 INFO:teuthology.orchestra.run.smithi136.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-20T01:19:34.148 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-20T01:19:34.148 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-20T01:19:34.148 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-20T01:19:34.148 INFO:teuthology.orchestra.run.smithi136.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-20T01:19:34.148 INFO:teuthology.orchestra.run.smithi136.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-20T01:19:34.149 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-20T01:19:34.149 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-20T01:19:34.149 INFO:teuthology.orchestra.run.smithi136.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-20T01:19:34.149 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-20T01:19:34.149 INFO:teuthology.orchestra.run.smithi136.stdout: python3-portend-2.6-1.el8.noarch 2023-07-20T01:19:34.149 INFO:teuthology.orchestra.run.smithi136.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-20T01:19:34.149 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-20T01:19:34.149 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-20T01:19:34.150 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-20T01:19:34.150 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-20T01:19:34.150 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-20T01:19:34.150 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:19:34.150 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:19:34.150 INFO:teuthology.orchestra.run.smithi136.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-20T01:19:34.150 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-20T01:19:34.150 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:19:34.151 INFO:teuthology.orchestra.run.smithi136.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-20T01:19:34.151 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-20T01:19:34.151 INFO:teuthology.orchestra.run.smithi136.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-20T01:19:34.151 INFO:teuthology.orchestra.run.smithi136.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-20T01:19:34.151 INFO:teuthology.orchestra.run.smithi136.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-20T01:19:34.151 INFO:teuthology.orchestra.run.smithi136.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-20T01:19:34.151 INFO:teuthology.orchestra.run.smithi136.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-20T01:19:34.151 INFO:teuthology.orchestra.run.smithi136.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-20T01:19:34.152 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-20T01:19:34.152 INFO:teuthology.orchestra.run.smithi136.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-20T01:19:34.152 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-20T01:19:34.152 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-20T01:19:34.152 INFO:teuthology.orchestra.run.smithi136.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-20T01:19:34.152 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-fuse-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:19:34.152 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-mirror-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:19:34.153 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-nbd-2:18.0.0-5040.g1b0b2eb0.el8.x86_64 2023-07-20T01:19:34.153 INFO:teuthology.orchestra.run.smithi136.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-20T01:19:34.153 INFO:teuthology.orchestra.run.smithi136.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-20T01:19:34.153 INFO:teuthology.orchestra.run.smithi136.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-20T01:19:34.153 INFO:teuthology.orchestra.run.smithi136.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-20T01:19:34.153 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-20T01:19:34.154 INFO:teuthology.orchestra.run.smithi136.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-20T01:19:34.154 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-07-20T01:19:34.154 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2023-07-20T01:19:34.331 DEBUG:teuthology.parallel:result is None 2023-07-20T01:19:34.332 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T01:19:34.686 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-20T01:19:34.716 INFO:teuthology.orchestra.run.smithi060.stdout:18.0.0-5040.g1b0b2eb0.el8 2023-07-20T01:19:34.717 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-5040.g1b0b2eb0.el8 2023-07-20T01:19:34.717 INFO:teuthology.task.install:The correct ceph version 18.0.0-5040.g1b0b2eb0 is installed. 2023-07-20T01:19:34.718 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T01:19:35.046 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-20T01:19:35.074 INFO:teuthology.orchestra.run.smithi136.stdout:18.0.0-5040.g1b0b2eb0.el8 2023-07-20T01:19:35.074 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-5040.g1b0b2eb0.el8 2023-07-20T01:19:35.075 INFO:teuthology.task.install:The correct ceph version 18.0.0-5040.g1b0b2eb0 is installed. 2023-07-20T01:19:35.075 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T01:19:35.366 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-20T01:19:35.395 INFO:teuthology.orchestra.run.smithi183.stdout:18.0.0-5040.g1b0b2eb0.el8 2023-07-20T01:19:35.396 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-5040.g1b0b2eb0.el8 2023-07-20T01:19:35.396 INFO:teuthology.task.install:The correct ceph version 18.0.0-5040.g1b0b2eb0 is installed. 2023-07-20T01:19:35.396 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-07-20T01:19:35.397 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-07-20T01:19:35.397 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-20T01:19:35.448 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-07-20T01:19:35.448 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-20T01:19:35.495 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2023-07-20T01:19:35.495 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-20T01:19:35.540 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-07-20T01:19:35.541 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-07-20T01:19:35.541 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/daemon-helper 2023-07-20T01:19:35.573 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-20T01:19:35.645 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-07-20T01:19:35.646 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/daemon-helper 2023-07-20T01:19:35.676 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-20T01:19:35.746 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2023-07-20T01:19:35.747 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/daemon-helper 2023-07-20T01:19:35.776 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-20T01:19:35.845 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-07-20T01:19:35.845 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-07-20T01:19:35.846 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-20T01:19:35.878 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-20T01:19:35.951 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-07-20T01:19:35.952 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-20T01:19:35.981 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-20T01:19:36.050 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2023-07-20T01:19:36.051 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-20T01:19:36.079 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-20T01:19:36.146 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-07-20T01:19:36.147 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-07-20T01:19:36.147 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/stdin-killer 2023-07-20T01:19:36.180 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-20T01:19:36.251 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-07-20T01:19:36.251 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/stdin-killer 2023-07-20T01:19:36.281 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-20T01:19:36.349 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2023-07-20T01:19:36.350 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/stdin-killer 2023-07-20T01:19:36.378 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-20T01:19:36.447 INFO:teuthology.run_tasks:Running task cephadm... 2023-07-20T01:19:36.537 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd heartbeat grace': 60, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'SLOW_OPS', 'slow request'], 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2023-07-20T01:19:36.538 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T01:19:36.547 INFO:tasks.cephadm:Cluster fsid is 7daf031e-269b-11ee-9b34-001a4aab830c 2023-07-20T01:19:36.548 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-07-20T01:19:36.548 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.60', 'mon.b': '172.21.15.136', 'mon.c': '172.21.15.183'} 2023-07-20T01:19:36.548 INFO:tasks.cephadm:First mon is mon.a on smithi060 2023-07-20T01:19:36.549 INFO:tasks.cephadm:First mgr is x 2023-07-20T01:19:36.549 INFO:tasks.cephadm:Normalizing hostnames... 2023-07-20T01:19:36.549 DEBUG:teuthology.orchestra.run.smithi060:> sudo hostname $(hostname -s) 2023-07-20T01:19:36.584 DEBUG:teuthology.orchestra.run.smithi136:> sudo hostname $(hostname -s) 2023-07-20T01:19:36.615 DEBUG:teuthology.orchestra.run.smithi183:> sudo hostname $(hostname -s) 2023-07-20T01:19:36.644 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2023-07-20T01:19:36.644 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T01:19:37.056 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '1b0b2eb005a7c51d97a788b9900d511e1394b537', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/71562/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.0.0-5040-g1b0b2eb0', 'node_name': '172.21.5.31+adami01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.0.0-5040.g1b0b2eb0'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2023-07-19 22:37:13.930734', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20230719.201220', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2023-07-20T01:19:37.416 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-pdonnell-testing-20230719.201220, sha1 1b0b2eb005a7c51d97a788b9900d511e1394b537 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=1b0b2eb005a7c51d97a788b9900d511e1394b537 2023-07-20T01:19:37.417 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/x86_64/flavors/default/cephadm 2023-07-20T01:19:37.418 DEBUG:teuthology.orchestra.run.smithi060:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-20T01:19:38.408 INFO:teuthology.orchestra.run.smithi060.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 20 01:19 /home/ubuntu/cephtest/cephadm 2023-07-20T01:19:38.408 DEBUG:teuthology.orchestra.run.smithi136:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-20T01:19:39.423 INFO:teuthology.orchestra.run.smithi136.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 20 01:19 /home/ubuntu/cephtest/cephadm 2023-07-20T01:19:39.423 DEBUG:teuthology.orchestra.run.smithi183:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230719.201220/1b0b2eb005a7c51d97a788b9900d511e1394b537/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-20T01:19:40.531 INFO:teuthology.orchestra.run.smithi183.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 20 01:19 /home/ubuntu/cephtest/cephadm 2023-07-20T01:19:40.531 DEBUG:teuthology.orchestra.run.smithi060:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-20T01:19:40.553 DEBUG:teuthology.orchestra.run.smithi136:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-20T01:19:40.576 DEBUG:teuthology.orchestra.run.smithi183:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-20T01:19:40.619 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 on all hosts... 2023-07-20T01:19:40.619 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 pull 2023-07-20T01:19:40.622 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 pull 2023-07-20T01:19:40.624 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 pull 2023-07-20T01:19:41.822 INFO:teuthology.orchestra.run.smithi183.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T01:19:41.872 INFO:teuthology.orchestra.run.smithi060.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T01:19:45.914 INFO:teuthology.orchestra.run.smithi136.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T01:20:15.095 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2023-07-20T01:20:15.095 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_version": "ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev)", 2023-07-20T01:20:15.095 INFO:teuthology.orchestra.run.smithi136.stdout: "image_id": "3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7", 2023-07-20T01:20:15.095 INFO:teuthology.orchestra.run.smithi136.stdout: "repo_digests": [ 2023-07-20T01:20:15.095 INFO:teuthology.orchestra.run.smithi136.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5e9e7fcbd408a2723f648774346bcaf016d0aaddd533517186bfd5cb559518b1" 2023-07-20T01:20:15.095 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2023-07-20T01:20:15.096 INFO:teuthology.orchestra.run.smithi136.stdout:} 2023-07-20T01:20:23.182 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2023-07-20T01:20:23.182 INFO:teuthology.orchestra.run.smithi183.stdout: "ceph_version": "ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev)", 2023-07-20T01:20:23.183 INFO:teuthology.orchestra.run.smithi183.stdout: "image_id": "3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7", 2023-07-20T01:20:23.183 INFO:teuthology.orchestra.run.smithi183.stdout: "repo_digests": [ 2023-07-20T01:20:23.183 INFO:teuthology.orchestra.run.smithi183.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5e9e7fcbd408a2723f648774346bcaf016d0aaddd533517186bfd5cb559518b1" 2023-07-20T01:20:23.183 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2023-07-20T01:20:23.183 INFO:teuthology.orchestra.run.smithi183.stdout:} 2023-07-20T01:20:28.760 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2023-07-20T01:20:28.761 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_version": "ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev)", 2023-07-20T01:20:28.761 INFO:teuthology.orchestra.run.smithi060.stdout: "image_id": "3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7", 2023-07-20T01:20:28.761 INFO:teuthology.orchestra.run.smithi060.stdout: "repo_digests": [ 2023-07-20T01:20:28.761 INFO:teuthology.orchestra.run.smithi060.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5e9e7fcbd408a2723f648774346bcaf016d0aaddd533517186bfd5cb559518b1" 2023-07-20T01:20:28.761 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2023-07-20T01:20:28.761 INFO:teuthology.orchestra.run.smithi060.stdout:} 2023-07-20T01:20:28.782 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /etc/ceph 2023-07-20T01:20:28.814 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /etc/ceph 2023-07-20T01:20:28.850 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /etc/ceph 2023-07-20T01:20:29.059 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 777 /etc/ceph 2023-07-20T01:20:29.088 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 777 /etc/ceph 2023-07-20T01:20:29.120 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 777 /etc/ceph 2023-07-20T01:20:29.151 INFO:tasks.cephadm:Writing seed config... 2023-07-20T01:20:29.152 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-07-20T01:20:29.153 INFO:tasks.cephadm: override: [client] debug client = 20 2023-07-20T01:20:29.153 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-07-20T01:20:29.153 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-07-20T01:20:29.153 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-07-20T01:20:29.153 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-07-20T01:20:29.154 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-07-20T01:20:29.154 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-07-20T01:20:29.154 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2023-07-20T01:20:29.154 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2023-07-20T01:20:29.154 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2023-07-20T01:20:29.155 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2023-07-20T01:20:29.155 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-07-20T01:20:29.155 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-07-20T01:20:29.155 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-07-20T01:20:29.156 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-07-20T01:20:29.156 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2023-07-20T01:20:29.156 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-07-20T01:20:29.156 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-07-20T01:20:29.156 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-07-20T01:20:29.157 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2023-07-20T01:20:29.157 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2023-07-20T01:20:29.157 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-07-20T01:20:29.157 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-07-20T01:20:29.157 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-07-20T01:20:29.158 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-07-20T01:20:29.158 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-07-20T01:20:29.158 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-07-20T01:20:29.158 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-07-20T01:20:29.158 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2023-07-20T01:20:29.159 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-07-20T01:20:29.159 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2023-07-20T01:20:29.159 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2023-07-20T01:20:29.159 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-07-20T01:20:29.159 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-07-20T01:20:29.160 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2023-07-20T01:20:29.160 INFO:tasks.cephadm: override: [osd] filestore flush min = 0 2023-07-20T01:20:29.160 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-07-20T01:20:29.160 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-07-20T01:20:29.160 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-07-20T01:20:29.161 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-07-20T01:20:29.161 INFO:tasks.cephadm: override: [osd] osd heartbeat grace = 60 2023-07-20T01:20:29.161 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-07-20T01:20:29.161 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-07-20T01:20:29.161 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2023-07-20T01:20:29.162 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2023-07-20T01:20:29.162 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-07-20T01:20:29.163 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-07-20T01:20:29.180 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 7daf031e-269b-11ee-9b34-001a4aab830c [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 filestore flush min = 0 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd heartbeat grace = 60 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-07-20T01:20:29.181 DEBUG:teuthology.orchestra.run.smithi060:mon.a> sudo journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mon.a.service 2023-07-20T01:20:29.222 DEBUG:teuthology.orchestra.run.smithi060:mgr.x> sudo journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mgr.x.service 2023-07-20T01:20:29.251 INFO:journalctl@ceph.mon.a.smithi060.stdout:-- Logs begin at Thu 2023-07-20 01:00:02 UTC. -- 2023-07-20T01:20:29.252 INFO:tasks.cephadm:Bootstrapping... 2023-07-20T01:20:29.252 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 -v bootstrap --fsid 7daf031e-269b-11ee-9b34-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.60 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-07-20T01:20:29.279 INFO:journalctl@ceph.mgr.x.smithi060.stdout:-- Logs begin at Thu 2023-07-20 01:00:02 UTC. -- 2023-07-20T01:20:29.419 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2023-07-20T01:20:29.419 INFO:teuthology.orchestra.run.smithi060.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537', '-v', 'bootstrap', '--fsid', '7daf031e-269b-11ee-9b34-001a4aab830c', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.60', '--skip-admin-label'] 2023-07-20T01:20:29.567 INFO:teuthology.orchestra.run.smithi060.stdout:/bin/podman: stdout 4.1.1 2023-07-20T01:20:29.568 INFO:teuthology.orchestra.run.smithi060.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-07-20T01:20:29.568 INFO:teuthology.orchestra.run.smithi060.stdout:Verifying podman|docker is present... 2023-07-20T01:20:29.718 INFO:teuthology.orchestra.run.smithi060.stdout:/bin/podman: stdout 4.1.1 2023-07-20T01:20:29.719 INFO:teuthology.orchestra.run.smithi060.stdout:Verifying lvm2 is present... 2023-07-20T01:20:29.719 INFO:teuthology.orchestra.run.smithi060.stdout:Verifying time synchronization is in place... 2023-07-20T01:20:29.726 INFO:teuthology.orchestra.run.smithi060.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-20T01:20:29.726 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-20T01:20:29.732 INFO:teuthology.orchestra.run.smithi060.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-20T01:20:29.732 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stdout inactive 2023-07-20T01:20:29.738 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stdout enabled 2023-07-20T01:20:29.744 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stdout active 2023-07-20T01:20:29.744 INFO:teuthology.orchestra.run.smithi060.stdout:Unit chronyd.service is enabled and running 2023-07-20T01:20:29.744 INFO:teuthology.orchestra.run.smithi060.stdout:Repeating the final host check... 2023-07-20T01:20:29.891 INFO:teuthology.orchestra.run.smithi060.stdout:/bin/podman: stdout 4.1.1 2023-07-20T01:20:29.892 INFO:teuthology.orchestra.run.smithi060.stdout:podman (/bin/podman) version 4.1.1 is present 2023-07-20T01:20:29.892 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl is present 2023-07-20T01:20:29.892 INFO:teuthology.orchestra.run.smithi060.stdout:lvcreate is present 2023-07-20T01:20:29.898 INFO:teuthology.orchestra.run.smithi060.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-20T01:20:29.898 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-20T01:20:29.903 INFO:teuthology.orchestra.run.smithi060.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-20T01:20:29.904 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stdout inactive 2023-07-20T01:20:29.909 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stdout enabled 2023-07-20T01:20:29.916 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stdout active 2023-07-20T01:20:29.916 INFO:teuthology.orchestra.run.smithi060.stdout:Unit chronyd.service is enabled and running 2023-07-20T01:20:29.916 INFO:teuthology.orchestra.run.smithi060.stdout:Host looks OK 2023-07-20T01:20:29.916 INFO:teuthology.orchestra.run.smithi060.stdout:Cluster fsid: 7daf031e-269b-11ee-9b34-001a4aab830c 2023-07-20T01:20:29.916 INFO:teuthology.orchestra.run.smithi060.stdout:Acquiring lock 140591722486248 on /run/cephadm/7daf031e-269b-11ee-9b34-001a4aab830c.lock 2023-07-20T01:20:29.917 INFO:teuthology.orchestra.run.smithi060.stdout:Lock 140591722486248 acquired on /run/cephadm/7daf031e-269b-11ee-9b34-001a4aab830c.lock 2023-07-20T01:20:29.917 INFO:teuthology.orchestra.run.smithi060.stdout:Verifying IP 172.21.15.60 port 3300 ... 2023-07-20T01:20:29.917 INFO:teuthology.orchestra.run.smithi060.stdout:Verifying IP 172.21.15.60 port 6789 ... 2023-07-20T01:20:29.917 INFO:teuthology.orchestra.run.smithi060.stdout:Base mon IP(s) is [172.21.15.60:3300, 172.21.15.60:6789], mon addrv is [v2:172.21.15.60:3300,v1:172.21.15.60:6789] 2023-07-20T01:20:29.920 INFO:teuthology.orchestra.run.smithi060.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp metric 100 2023-07-20T01:20:29.921 INFO:teuthology.orchestra.run.smithi060.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.60 metric 100 2023-07-20T01:20:29.923 INFO:teuthology.orchestra.run.smithi060.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-07-20T01:20:29.923 INFO:teuthology.orchestra.run.smithi060.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2023-07-20T01:20:29.923 INFO:teuthology.orchestra.run.smithi060.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1795sec hoplimit 64 pref medium 2023-07-20T01:20:29.926 INFO:teuthology.orchestra.run.smithi060.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-07-20T01:20:29.926 INFO:teuthology.orchestra.run.smithi060.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-07-20T01:20:29.926 INFO:teuthology.orchestra.run.smithi060.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-20T01:20:29.926 INFO:teuthology.orchestra.run.smithi060.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2023-07-20T01:20:29.926 INFO:teuthology.orchestra.run.smithi060.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1560/64 scope link 2023-07-20T01:20:29.926 INFO:teuthology.orchestra.run.smithi060.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-20T01:20:29.927 INFO:teuthology.orchestra.run.smithi060.stdout:Mon IP `172.21.15.60` is in CIDR network `172.21.0.0/20` 2023-07-20T01:20:29.927 INFO:teuthology.orchestra.run.smithi060.stdout:Mon IP `172.21.15.60` is in CIDR network `172.21.0.0/20` 2023-07-20T01:20:29.927 INFO:teuthology.orchestra.run.smithi060.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-07-20T01:20:29.928 INFO:teuthology.orchestra.run.smithi060.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-07-20T01:20:29.928 INFO:teuthology.orchestra.run.smithi060.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T01:20:30.372 INFO:teuthology.orchestra.run.smithi060.stdout:/bin/podman: stdout 3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7 2023-07-20T01:20:30.372 INFO:teuthology.orchestra.run.smithi060.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537... 2023-07-20T01:20:30.372 INFO:teuthology.orchestra.run.smithi060.stdout:/bin/podman: stderr Getting image source signatures 2023-07-20T01:20:30.372 INFO:teuthology.orchestra.run.smithi060.stdout:/bin/podman: stderr Copying blob sha256:9a3af2d1f966fade87cf08b715899d7eff2dde16dabc6e2e4c3ec2059da8adc1 2023-07-20T01:20:30.373 INFO:teuthology.orchestra.run.smithi060.stdout:/bin/podman: stderr Copying blob sha256:6022c611d474f2b306ddd1edfc8360123ece5266119096f25fde843d7967e636 2023-07-20T01:20:30.373 INFO:teuthology.orchestra.run.smithi060.stdout:/bin/podman: stderr Copying config sha256:3fb3b23259e6e12c172498efe90efa537c0b5171b4b834ce7c8a2ef97ba82ca7 2023-07-20T01:20:30.373 INFO:teuthology.orchestra.run.smithi060.stdout:/bin/podman: stderr Writing manifest to image destination 2023-07-20T01:20:30.373 INFO:teuthology.orchestra.run.smithi060.stdout:/bin/podman: stderr Storing signatures 2023-07-20T01:20:31.219 INFO:teuthology.orchestra.run.smithi060.stdout:ceph: stdout ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev) 2023-07-20T01:20:31.220 INFO:teuthology.orchestra.run.smithi060.stdout:Ceph version: ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev) 2023-07-20T01:20:31.220 INFO:teuthology.orchestra.run.smithi060.stdout:Extracting ceph user uid/gid from container image... 2023-07-20T01:20:31.969 INFO:teuthology.orchestra.run.smithi060.stdout:stat: stdout 167 167 2023-07-20T01:20:31.969 INFO:teuthology.orchestra.run.smithi060.stdout:Creating initial keys... 2023-07-20T01:20:32.829 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph-authtool: stdout AQDgi7hkGvCMEhAAAwbgZOD4cFoK8H2Fxx+BuQ== 2023-07-20T01:20:33.480 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph-authtool: stdout AQDhi7hk+kcUCxAAxRSjYoWmdCY04+4sfQx3nA== 2023-07-20T01:20:34.227 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph-authtool: stdout AQDhi7hk09JWMxAAPQvhN4fve4kXb9clwcv4SA== 2023-07-20T01:20:34.228 INFO:teuthology.orchestra.run.smithi060.stdout:Creating initial monmap... 2023-07-20T01:20:34.974 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-20T01:20:34.974 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2023-07-20T01:20:34.974 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 7daf031e-269b-11ee-9b34-001a4aab830c 2023-07-20T01:20:34.974 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-20T01:20:34.974 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool for a [v2:172.21.15.60:3300,v1:172.21.15.60:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-20T01:20:34.975 INFO:teuthology.orchestra.run.smithi060.stdout:setting min_mon_release = pacific 2023-07-20T01:20:34.975 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/monmaptool: set fsid to 7daf031e-269b-11ee-9b34-001a4aab830c 2023-07-20T01:20:34.975 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-20T01:20:34.975 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:20:34.975 INFO:teuthology.orchestra.run.smithi060.stdout:Creating mon... 2023-07-20T01:20:35.728 INFO:teuthology.orchestra.run.smithi060.stdout:create mon.a on 2023-07-20T01:20:35.862 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-07-20T01:20:35.954 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-07-20T01:20:36.081 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-7daf031e-269b-11ee-9b34-001a4aab830c.target → /etc/systemd/system/ceph-7daf031e-269b-11ee-9b34-001a4aab830c.target. 2023-07-20T01:20:36.081 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-7daf031e-269b-11ee-9b34-001a4aab830c.target → /etc/systemd/system/ceph-7daf031e-269b-11ee-9b34-001a4aab830c.target. 2023-07-20T01:20:36.259 INFO:teuthology.orchestra.run.smithi060.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mon.a 2023-07-20T01:20:36.259 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stderr Failed to reset failed state of unit ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mon.a.service: Unit ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mon.a.service not loaded. 2023-07-20T01:20:36.353 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-7daf031e-269b-11ee-9b34-001a4aab830c.target.wants/ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mon.a.service → /etc/systemd/system/ceph-7daf031e-269b-11ee-9b34-001a4aab830c@.service. 2023-07-20T01:20:36.668 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:36 smithi060 systemd[1]: Starting Ceph mon.a for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:20:36.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:36 smithi060 podman[95975]: 2023-07-20T01:20:36.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:36 smithi060 bash[95975]: db10377c45da77fe220af6022a39d384e786292dab1f2f8ff9a66261a26a5c89 2023-07-20T01:20:36.959 INFO:teuthology.orchestra.run.smithi060.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-20T01:20:36.960 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stdout disabled 2023-07-20T01:20:36.967 INFO:teuthology.orchestra.run.smithi060.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-20T01:20:36.967 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stdout inactive 2023-07-20T01:20:36.967 INFO:teuthology.orchestra.run.smithi060.stdout:firewalld.service is not enabled 2023-07-20T01:20:36.967 INFO:teuthology.orchestra.run.smithi060.stdout:Not possible to enable service . firewalld.service is not available 2023-07-20T01:20:36.968 INFO:teuthology.orchestra.run.smithi060.stdout:Waiting for mon to start... 2023-07-20T01:20:36.968 INFO:teuthology.orchestra.run.smithi060.stdout:Waiting for mon... 2023-07-20T01:20:37.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:36 smithi060 ceph-mon[96014]: mkfs 7daf031e-269b-11ee-9b34-001a4aab830c 2023-07-20T01:20:37.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:36 smithi060 ceph-mon[96014]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-20T01:20:37.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:36 smithi060 systemd[1]: Started Ceph mon.a for 7daf031e-269b-11ee-9b34-001a4aab830c. 2023-07-20T01:20:37.912 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout cluster: 2023-07-20T01:20:37.916 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout id: 7daf031e-269b-11ee-9b34-001a4aab830c 2023-07-20T01:20:37.917 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-07-20T01:20:37.917 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 2023-07-20T01:20:37.917 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout services: 2023-07-20T01:20:37.917 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.619945s) 2023-07-20T01:20:37.917 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-07-20T01:20:37.917 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-07-20T01:20:37.918 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 2023-07-20T01:20:37.918 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout data: 2023-07-20T01:20:37.918 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-07-20T01:20:37.918 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-07-20T01:20:37.918 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-07-20T01:20:37.918 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout pgs: 2023-07-20T01:20:37.918 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 2023-07-20T01:20:37.918 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.420+0000 7feb696b6700 1 Processor -- start 2023-07-20T01:20:37.919 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.420+0000 7feb696b6700 1 -- start start 2023-07-20T01:20:37.919 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.421+0000 7feb696b6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb64105fd0 0x7feb641063f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:20:37.919 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.421+0000 7feb696b6700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb641069c0 con 0x7feb64105fd0 2023-07-20T01:20:37.919 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.421+0000 7feb62ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb64105fd0 0x7feb641063f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:20:37.919 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.421+0000 7feb62ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb64105fd0 0x7feb641063f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41774/0 (socket says 172.21.15.60:41774) 2023-07-20T01:20:37.919 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.421+0000 7feb62ffd700 1 -- 172.21.15.60:0/2246086384 learned_addr learned my addr 172.21.15.60:0/2246086384 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:20:37.919 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.422+0000 7feb62ffd700 1 -- 172.21.15.60:0/2246086384 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb641071d0 con 0x7feb64105fd0 2023-07-20T01:20:37.920 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.422+0000 7feb62ffd700 1 --2- 172.21.15.60:0/2246086384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb64105fd0 0x7feb641063f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7feb540098a0 tx=0x7feb54009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8be35593c4b0310 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:20:37.920 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.422+0000 7feb61ffb700 1 -- 172.21.15.60:0/2246086384 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb54004160 con 0x7feb64105fd0 2023-07-20T01:20:37.920 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.422+0000 7feb61ffb700 1 -- 172.21.15.60:0/2246086384 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb540042c0 con 0x7feb64105fd0 2023-07-20T01:20:37.920 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.422+0000 7feb61ffb700 1 -- 172.21.15.60:0/2246086384 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb540045a0 con 0x7feb64105fd0 2023-07-20T01:20:37.920 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.422+0000 7feb696b6700 1 -- 172.21.15.60:0/2246086384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb64105fd0 msgr2=0x7feb641063f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:20:37.920 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.422+0000 7feb696b6700 1 --2- 172.21.15.60:0/2246086384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb64105fd0 0x7feb641063f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7feb540098a0 tx=0x7feb54009bb0 comp rx=0 tx=0).stop 2023-07-20T01:20:37.920 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.423+0000 7feb696b6700 1 -- 172.21.15.60:0/2246086384 shutdown_connections 2023-07-20T01:20:37.920 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.423+0000 7feb696b6700 1 --2- 172.21.15.60:0/2246086384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb64105fd0 0x7feb641063f0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:20:37.921 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.423+0000 7feb696b6700 1 -- 172.21.15.60:0/2246086384 >> 172.21.15.60:0/2246086384 conn(0x7feb641015b0 msgr2=0x7feb641039d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:20:37.921 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.423+0000 7feb696b6700 1 -- 172.21.15.60:0/2246086384 shutdown_connections 2023-07-20T01:20:37.921 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.423+0000 7feb696b6700 1 -- 172.21.15.60:0/2246086384 wait complete. 2023-07-20T01:20:37.921 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.423+0000 7feb696b6700 1 Processor -- start 2023-07-20T01:20:37.921 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.424+0000 7feb696b6700 1 -- start start 2023-07-20T01:20:37.921 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.424+0000 7feb696b6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb64105fd0 0x7feb64199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:20:37.921 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.424+0000 7feb696b6700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb64199a60 con 0x7feb64105fd0 2023-07-20T01:20:37.922 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.424+0000 7feb62ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb64105fd0 0x7feb64199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:20:37.922 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.424+0000 7feb62ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb64105fd0 0x7feb64199520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41776/0 (socket says 172.21.15.60:41776) 2023-07-20T01:20:37.922 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.424+0000 7feb62ffd700 1 -- 172.21.15.60:0/3421759995 learned_addr learned my addr 172.21.15.60:0/3421759995 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:20:37.922 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.424+0000 7feb62ffd700 1 -- 172.21.15.60:0/3421759995 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb54009550 con 0x7feb64105fd0 2023-07-20T01:20:37.922 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.424+0000 7feb62ffd700 1 --2- 172.21.15.60:0/3421759995 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb64105fd0 0x7feb64199520 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7feb54004000 tx=0x7feb54004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:20:37.922 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.425+0000 7feb5bfff700 1 -- 172.21.15.60:0/3421759995 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb54011630 con 0x7feb64105fd0 2023-07-20T01:20:37.922 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.425+0000 7feb5bfff700 1 -- 172.21.15.60:0/3421759995 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb54011790 con 0x7feb64105fd0 2023-07-20T01:20:37.922 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.425+0000 7feb696b6700 1 -- 172.21.15.60:0/3421759995 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb64199c60 con 0x7feb64105fd0 2023-07-20T01:20:37.923 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.425+0000 7feb5bfff700 1 -- 172.21.15.60:0/3421759995 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb54011a30 con 0x7feb64105fd0 2023-07-20T01:20:37.923 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.425+0000 7feb696b6700 1 -- 172.21.15.60:0/3421759995 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb6419a080 con 0x7feb64105fd0 2023-07-20T01:20:37.923 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.425+0000 7feb5bfff700 1 -- 172.21.15.60:0/3421759995 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7feb54022070 con 0x7feb64105fd0 2023-07-20T01:20:37.923 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.425+0000 7feb5bfff700 1 -- 172.21.15.60:0/3421759995 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7feb5401c790 con 0x7feb64105fd0 2023-07-20T01:20:37.923 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.426+0000 7feb696b6700 1 -- 172.21.15.60:0/3421759995 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb64192dd0 con 0x7feb64105fd0 2023-07-20T01:20:37.923 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.428+0000 7feb5bfff700 1 -- 172.21.15.60:0/3421759995 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7feb5401bd40 con 0x7feb64105fd0 2023-07-20T01:20:37.923 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.561+0000 7feb696b6700 1 -- 172.21.15.60:0/3421759995 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7feb64061da0 con 0x7feb64105fd0 2023-07-20T01:20:37.923 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.561+0000 7feb5bfff700 1 -- 172.21.15.60:0/3421759995 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7feb5401b450 con 0x7feb64105fd0 2023-07-20T01:20:37.924 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.562+0000 7feb696b6700 1 -- 172.21.15.60:0/3421759995 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb64105fd0 msgr2=0x7feb64199520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:20:37.924 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.562+0000 7feb696b6700 1 --2- 172.21.15.60:0/3421759995 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb64105fd0 0x7feb64199520 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7feb54004000 tx=0x7feb54004750 comp rx=0 tx=0).stop 2023-07-20T01:20:37.924 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.562+0000 7feb696b6700 1 -- 172.21.15.60:0/3421759995 shutdown_connections 2023-07-20T01:20:37.924 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.562+0000 7feb696b6700 1 --2- 172.21.15.60:0/3421759995 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb64105fd0 0x7feb64199520 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:20:37.924 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.562+0000 7feb696b6700 1 -- 172.21.15.60:0/3421759995 >> 172.21.15.60:0/3421759995 conn(0x7feb641015b0 msgr2=0x7feb6418d9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:20:37.924 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.562+0000 7feb696b6700 1 -- 172.21.15.60:0/3421759995 shutdown_connections 2023-07-20T01:20:37.924 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:37.562+0000 7feb696b6700 1 -- 172.21.15.60:0/3421759995 wait complete. 2023-07-20T01:20:37.925 INFO:teuthology.orchestra.run.smithi060.stdout:mon is available 2023-07-20T01:20:37.925 INFO:teuthology.orchestra.run.smithi060.stdout:Assimilating anything we can from ceph.conf... 2023-07-20T01:20:38.151 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:37 smithi060 ceph-mon[96014]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-20T01:20:38.151 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:37 smithi060 ceph-mon[96014]: monmap e1: 1 mons at {a=[v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0]} removed_ranks: {} 2023-07-20T01:20:38.151 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:37 smithi060 ceph-mon[96014]: fsmap 2023-07-20T01:20:38.151 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:37 smithi060 ceph-mon[96014]: osdmap e1: 0 total, 0 up, 0 in 2023-07-20T01:20:38.151 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:37 smithi060 ceph-mon[96014]: mgrmap e1: no daemons active 2023-07-20T01:20:38.152 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:37 smithi060 ceph-mon[96014]: from='client.? 172.21.15.60:0/3421759995' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-07-20T01:20:38.963 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 2023-07-20T01:20:38.964 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout [global] 2023-07-20T01:20:38.964 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout fsid = 7daf031e-269b-11ee-9b34-001a4aab830c 2023-07-20T01:20:38.964 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.60:3300,v1:172.21.15.60:6789] 2023-07-20T01:20:38.964 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-20T01:20:38.965 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-20T01:20:38.965 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-20T01:20:38.965 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-20T01:20:38.965 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 2023-07-20T01:20:38.965 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-20T01:20:38.965 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-20T01:20:38.965 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 2023-07-20T01:20:38.966 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout [osd] 2023-07-20T01:20:38.966 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2023-07-20T01:20:38.966 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-07-20T01:20:38.966 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-07-20T01:20:38.966 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-07-20T01:20:38.966 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-20T01:20:38.967 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-20T01:20:38.967 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.423+0000 7fa836aa3700 1 Processor -- start 2023-07-20T01:20:38.967 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.423+0000 7fa836aa3700 1 -- start start 2023-07-20T01:20:38.967 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.423+0000 7fa836aa3700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa830106a60 0x7fa830108e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:20:38.967 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.423+0000 7fa836aa3700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa830073ec0 con 0x7fa830106a60 2023-07-20T01:20:38.967 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.423+0000 7fa83483f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa830106a60 0x7fa830108e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:20:38.967 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.423+0000 7fa83483f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa830106a60 0x7fa830108e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41778/0 (socket says 172.21.15.60:41778) 2023-07-20T01:20:38.967 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.423+0000 7fa83483f700 1 -- 172.21.15.60:0/1052419381 learned_addr learned my addr 172.21.15.60:0/1052419381 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:20:38.968 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.424+0000 7fa83483f700 1 -- 172.21.15.60:0/1052419381 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa830109390 con 0x7fa830106a60 2023-07-20T01:20:38.968 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.424+0000 7fa83483f700 1 --2- 172.21.15.60:0/1052419381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa830106a60 0x7fa830108e50 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa8240098a0 tx=0x7fa824009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72e0d19e40427166 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:20:38.968 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.424+0000 7fa82f7fe700 1 -- 172.21.15.60:0/1052419381 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa824004160 con 0x7fa830106a60 2023-07-20T01:20:38.968 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.424+0000 7fa82f7fe700 1 -- 172.21.15.60:0/1052419381 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8240042c0 con 0x7fa830106a60 2023-07-20T01:20:38.968 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.424+0000 7fa82f7fe700 1 -- 172.21.15.60:0/1052419381 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa8240045a0 con 0x7fa830106a60 2023-07-20T01:20:38.968 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.425+0000 7fa836aa3700 1 -- 172.21.15.60:0/1052419381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa830106a60 msgr2=0x7fa830108e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:20:38.968 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.425+0000 7fa836aa3700 1 --2- 172.21.15.60:0/1052419381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa830106a60 0x7fa830108e50 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa8240098a0 tx=0x7fa824009bb0 comp rx=0 tx=0).stop 2023-07-20T01:20:38.969 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.425+0000 7fa836aa3700 1 -- 172.21.15.60:0/1052419381 shutdown_connections 2023-07-20T01:20:38.969 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.425+0000 7fa836aa3700 1 --2- 172.21.15.60:0/1052419381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa830106a60 0x7fa830108e50 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:20:38.969 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.425+0000 7fa836aa3700 1 -- 172.21.15.60:0/1052419381 >> 172.21.15.60:0/1052419381 conn(0x7fa830100490 msgr2=0x7fa8301028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:20:38.969 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.425+0000 7fa836aa3700 1 -- 172.21.15.60:0/1052419381 shutdown_connections 2023-07-20T01:20:38.969 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.425+0000 7fa836aa3700 1 -- 172.21.15.60:0/1052419381 wait complete. 2023-07-20T01:20:38.969 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.425+0000 7fa836aa3700 1 Processor -- start 2023-07-20T01:20:38.969 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.425+0000 7fa836aa3700 1 -- start start 2023-07-20T01:20:38.969 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.425+0000 7fa836aa3700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa830106a60 0x7fa830199500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:20:38.970 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.425+0000 7fa836aa3700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa830199a40 con 0x7fa830106a60 2023-07-20T01:20:38.970 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.426+0000 7fa83483f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa830106a60 0x7fa830199500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:20:38.970 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.426+0000 7fa83483f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa830106a60 0x7fa830199500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41780/0 (socket says 172.21.15.60:41780) 2023-07-20T01:20:38.970 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.426+0000 7fa83483f700 1 -- 172.21.15.60:0/1903374727 learned_addr learned my addr 172.21.15.60:0/1903374727 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:20:38.970 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.426+0000 7fa83483f700 1 -- 172.21.15.60:0/1903374727 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa824009550 con 0x7fa830106a60 2023-07-20T01:20:38.971 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.426+0000 7fa83483f700 1 --2- 172.21.15.60:0/1903374727 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa830106a60 0x7fa830199500 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa830073300 tx=0x7fa824004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:20:38.971 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.426+0000 7fa82dffb700 1 -- 172.21.15.60:0/1903374727 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa824011630 con 0x7fa830106a60 2023-07-20T01:20:38.971 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.426+0000 7fa82dffb700 1 -- 172.21.15.60:0/1903374727 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa824011790 con 0x7fa830106a60 2023-07-20T01:20:38.971 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.426+0000 7fa836aa3700 1 -- 172.21.15.60:0/1903374727 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa830199c40 con 0x7fa830106a60 2023-07-20T01:20:38.971 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.426+0000 7fa82dffb700 1 -- 172.21.15.60:0/1903374727 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa824011a30 con 0x7fa830106a60 2023-07-20T01:20:38.971 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.426+0000 7fa836aa3700 1 -- 172.21.15.60:0/1903374727 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa83019a060 con 0x7fa830106a60 2023-07-20T01:20:38.971 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.427+0000 7fa82dffb700 1 -- 172.21.15.60:0/1903374727 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa824022070 con 0x7fa830106a60 2023-07-20T01:20:38.972 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.427+0000 7fa82dffb700 1 -- 172.21.15.60:0/1903374727 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa82401c450 con 0x7fa830106a60 2023-07-20T01:20:38.972 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.428+0000 7fa836aa3700 1 -- 172.21.15.60:0/1903374727 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8301932b0 con 0x7fa830106a60 2023-07-20T01:20:38.972 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.430+0000 7fa82dffb700 1 -- 172.21.15.60:0/1903374727 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fa82401b450 con 0x7fa830106a60 2023-07-20T01:20:38.972 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.556+0000 7fa836aa3700 1 -- 172.21.15.60:0/1903374727 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fa830061da0 con 0x7fa830106a60 2023-07-20T01:20:38.972 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.559+0000 7fa82dffb700 1 -- 172.21.15.60:0/1903374727 <== mon.0 v2:172.21.15.60:3300/0 7 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fa82401b5f0 con 0x7fa830106a60 2023-07-20T01:20:38.972 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.559+0000 7fa82dffb700 1 -- 172.21.15.60:0/1903374727 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+454 (secure 0 0 0) 0x7fa824024560 con 0x7fa830106a60 2023-07-20T01:20:38.972 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.560+0000 7fa836aa3700 1 -- 172.21.15.60:0/1903374727 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa830106a60 msgr2=0x7fa830199500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:20:38.973 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.560+0000 7fa836aa3700 1 --2- 172.21.15.60:0/1903374727 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa830106a60 0x7fa830199500 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa830073300 tx=0x7fa824004750 comp rx=0 tx=0).stop 2023-07-20T01:20:38.973 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.560+0000 7fa836aa3700 1 -- 172.21.15.60:0/1903374727 shutdown_connections 2023-07-20T01:20:38.973 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.560+0000 7fa836aa3700 1 --2- 172.21.15.60:0/1903374727 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa830106a60 0x7fa830199500 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:20:38.973 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.560+0000 7fa836aa3700 1 -- 172.21.15.60:0/1903374727 >> 172.21.15.60:0/1903374727 conn(0x7fa830100490 msgr2=0x7fa830101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:20:38.973 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.561+0000 7fa836aa3700 1 -- 172.21.15.60:0/1903374727 shutdown_connections 2023-07-20T01:20:38.973 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:38.561+0000 7fa836aa3700 1 -- 172.21.15.60:0/1903374727 wait complete. 2023-07-20T01:20:38.973 INFO:teuthology.orchestra.run.smithi060.stdout:Generating new minimal ceph.conf... 2023-07-20T01:20:39.190 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:38 smithi060 ceph-mon[96014]: from='client.? 172.21.15.60:0/1903374727' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-20T01:20:39.190 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:38 smithi060 ceph-mon[96014]: from='client.? 172.21.15.60:0/1903374727' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-07-20T01:20:39.907 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.453+0000 7f3c0786e700 1 Processor -- start 2023-07-20T01:20:39.907 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.453+0000 7f3c0786e700 1 -- start start 2023-07-20T01:20:39.907 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.453+0000 7f3c0786e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c00105fb0 0x7f3c001063d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:20:39.908 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.453+0000 7f3c0786e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c001069a0 con 0x7f3c00105fb0 2023-07-20T01:20:39.908 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.453+0000 7f3c0560a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c00105fb0 0x7f3c001063d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:20:39.908 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.453+0000 7f3c0560a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c00105fb0 0x7f3c001063d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41782/0 (socket says 172.21.15.60:41782) 2023-07-20T01:20:39.908 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.453+0000 7f3c0560a700 1 -- 172.21.15.60:0/4254364687 learned_addr learned my addr 172.21.15.60:0/4254364687 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:20:39.908 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.454+0000 7f3c0560a700 1 -- 172.21.15.60:0/4254364687 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c001071b0 con 0x7f3c00105fb0 2023-07-20T01:20:39.908 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.454+0000 7f3c0560a700 1 --2- 172.21.15.60:0/4254364687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c00105fb0 0x7f3c001063d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3bf00098a0 tx=0x7f3bf0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f3b782435ac2c36 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:20:39.908 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.454+0000 7f3bf7fff700 1 -- 172.21.15.60:0/4254364687 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3bf0004160 con 0x7f3c00105fb0 2023-07-20T01:20:39.909 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.454+0000 7f3bf7fff700 1 -- 172.21.15.60:0/4254364687 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f3bf00042c0 con 0x7f3c00105fb0 2023-07-20T01:20:39.909 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.454+0000 7f3bf7fff700 1 -- 172.21.15.60:0/4254364687 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3bf0011940 con 0x7f3c00105fb0 2023-07-20T01:20:39.909 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.454+0000 7f3c0786e700 1 -- 172.21.15.60:0/4254364687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c00105fb0 msgr2=0x7f3c001063d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:20:39.909 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.454+0000 7f3c0786e700 1 --2- 172.21.15.60:0/4254364687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c00105fb0 0x7f3c001063d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3bf00098a0 tx=0x7f3bf0009bb0 comp rx=0 tx=0).stop 2023-07-20T01:20:39.909 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.454+0000 7f3c0786e700 1 -- 172.21.15.60:0/4254364687 shutdown_connections 2023-07-20T01:20:39.909 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.454+0000 7f3c0786e700 1 --2- 172.21.15.60:0/4254364687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c00105fb0 0x7f3c001063d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:20:39.909 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.455+0000 7f3c0786e700 1 -- 172.21.15.60:0/4254364687 >> 172.21.15.60:0/4254364687 conn(0x7f3c00101590 msgr2=0x7f3c001039b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:20:39.909 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.455+0000 7f3c0786e700 1 -- 172.21.15.60:0/4254364687 shutdown_connections 2023-07-20T01:20:39.910 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.455+0000 7f3c0786e700 1 -- 172.21.15.60:0/4254364687 wait complete. 2023-07-20T01:20:39.910 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.455+0000 7f3c0786e700 1 Processor -- start 2023-07-20T01:20:39.910 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.455+0000 7f3c0786e700 1 -- start start 2023-07-20T01:20:39.910 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.455+0000 7f3c0786e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c00105fb0 0x7f3c00199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:20:39.910 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.455+0000 7f3c0786e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0019a160 con 0x7f3c00105fb0 2023-07-20T01:20:39.910 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.456+0000 7f3c0560a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c00105fb0 0x7f3c00199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:20:39.910 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.456+0000 7f3c0560a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c00105fb0 0x7f3c00199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41784/0 (socket says 172.21.15.60:41784) 2023-07-20T01:20:39.911 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.456+0000 7f3c0560a700 1 -- 172.21.15.60:0/3793678977 learned_addr learned my addr 172.21.15.60:0/3793678977 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:20:39.911 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.456+0000 7f3c0560a700 1 -- 172.21.15.60:0/3793678977 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf0009550 con 0x7f3c00105fb0 2023-07-20T01:20:39.911 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.456+0000 7f3c0560a700 1 --2- 172.21.15.60:0/3793678977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c00105fb0 0x7f3c00199c20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0006af0 tx=0x7f3bf0011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:20:39.911 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.456+0000 7f3bf67fc700 1 -- 172.21.15.60:0/3793678977 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3bf0003bf0 con 0x7f3c00105fb0 2023-07-20T01:20:39.911 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.456+0000 7f3bf67fc700 1 -- 172.21.15.60:0/3793678977 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f3bf001b8b0 con 0x7f3c00105fb0 2023-07-20T01:20:39.911 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.456+0000 7f3c0786e700 1 -- 172.21.15.60:0/3793678977 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c0019a360 con 0x7f3c00105fb0 2023-07-20T01:20:39.911 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.456+0000 7f3bf67fc700 1 -- 172.21.15.60:0/3793678977 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3bf00198b0 con 0x7f3c00105fb0 2023-07-20T01:20:39.912 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.456+0000 7f3c0786e700 1 -- 172.21.15.60:0/3793678977 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c0019a780 con 0x7f3c00105fb0 2023-07-20T01:20:39.912 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.457+0000 7f3bf67fc700 1 -- 172.21.15.60:0/3793678977 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f3bf00233f0 con 0x7f3c00105fb0 2023-07-20T01:20:39.912 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.457+0000 7f3bf67fc700 1 -- 172.21.15.60:0/3793678977 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3bf001a750 con 0x7f3c00105fb0 2023-07-20T01:20:39.912 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.457+0000 7f3c0786e700 1 -- 172.21.15.60:0/3793678977 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c00061da0 con 0x7f3c00105fb0 2023-07-20T01:20:39.912 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.460+0000 7f3bf67fc700 1 -- 172.21.15.60:0/3793678977 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f3bf0028060 con 0x7f3c00105fb0 2023-07-20T01:20:39.912 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.587+0000 7f3c0786e700 1 -- 172.21.15.60:0/3793678977 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f3c0019d750 con 0x7f3c00105fb0 2023-07-20T01:20:39.912 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.587+0000 7f3bf67fc700 1 -- 172.21.15.60:0/3793678977 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f3bf001bd70 con 0x7f3c00105fb0 2023-07-20T01:20:39.913 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.588+0000 7f3c0786e700 1 -- 172.21.15.60:0/3793678977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c00105fb0 msgr2=0x7f3c00199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:20:39.913 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.588+0000 7f3c0786e700 1 --2- 172.21.15.60:0/3793678977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c00105fb0 0x7f3c00199c20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0006af0 tx=0x7f3bf0011a30 comp rx=0 tx=0).stop 2023-07-20T01:20:39.913 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.588+0000 7f3c0786e700 1 -- 172.21.15.60:0/3793678977 shutdown_connections 2023-07-20T01:20:39.913 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.588+0000 7f3c0786e700 1 --2- 172.21.15.60:0/3793678977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3c00105fb0 0x7f3c00199c20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:20:39.913 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.588+0000 7f3c0786e700 1 -- 172.21.15.60:0/3793678977 >> 172.21.15.60:0/3793678977 conn(0x7f3c00101590 msgr2=0x7f3c00102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:20:39.913 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.588+0000 7f3c0786e700 1 -- 172.21.15.60:0/3793678977 shutdown_connections 2023-07-20T01:20:39.913 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:39.588+0000 7f3c0786e700 1 -- 172.21.15.60:0/3793678977 wait complete. 2023-07-20T01:20:39.914 INFO:teuthology.orchestra.run.smithi060.stdout:Restarting the monitor... 2023-07-20T01:20:40.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:39 smithi060 systemd[1]: Stopping Ceph mon.a for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:20:40.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:39 smithi060 ceph-mon[96014]: from='client.? 172.21.15.60:0/3793678977' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:20:40.484 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:40 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mon-a[95990]: 2023-07-20T01:20:40.249+0000 7f2ad6018700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:20:40.485 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:40 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mon-a[95990]: 2023-07-20T01:20:40.249+0000 7f2ad6018700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-07-20T01:20:40.813 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:40 smithi060 bash[96352]: ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mon-a 2023-07-20T01:20:41.093 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:40 smithi060 systemd[1]: ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mon.a.service: Succeeded. 2023-07-20T01:20:41.094 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:40 smithi060 systemd[1]: Stopped Ceph mon.a for 7daf031e-269b-11ee-9b34-001a4aab830c. 2023-07-20T01:20:41.094 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:40 smithi060 systemd[1]: Starting Ceph mon.a for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:20:41.301 INFO:teuthology.orchestra.run.smithi060.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2023-07-20T01:20:41.354 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 podman[96483]: 2023-07-20T01:20:41.354 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: set uid:gid to 167:167 (ceph:ceph) 2023-07-20T01:20:41.354 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev), process ceph-mon, pid 8 2023-07-20T01:20:41.354 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: pidfile_write: ignore empty --pid-file 2023-07-20T01:20:41.354 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 bash[96483]: 6eb1bab930da6e0174125475ae8d64e07c2476c288c19b20bdaed67390f5c2c7 2023-07-20T01:20:41.354 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: load: jerasure load: lrc 2023-07-20T01:20:41.355 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: RocksDB version: 7.9.2 2023-07-20T01:20:41.355 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Git sha 0 2023-07-20T01:20:41.355 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Compile date 2023-07-19 21:36:35 2023-07-20T01:20:41.355 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: DB SUMMARY 2023-07-20T01:20:41.355 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: DB Session ID: 6W3497VOHB4T98ZDWLCP 2023-07-20T01:20:41.355 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: CURRENT file: CURRENT 2023-07-20T01:20:41.355 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: IDENTITY file: IDENTITY 2023-07-20T01:20:41.356 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2023-07-20T01:20:41.356 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2023-07-20T01:20:41.356 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 99155 ; 2023-07-20T01:20:41.356 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.error_if_exists: 0 2023-07-20T01:20:41.356 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.create_if_missing: 0 2023-07-20T01:20:41.356 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.paranoid_checks: 1 2023-07-20T01:20:41.356 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-20T01:20:41.357 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-20T01:20:41.357 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-20T01:20:41.357 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.env: 0x55b936d87c60 2023-07-20T01:20:41.357 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.fs: PosixFileSystem 2023-07-20T01:20:41.357 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.info_log: 0x55b938da72c0 2023-07-20T01:20:41.357 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.max_file_opening_threads: 16 2023-07-20T01:20:41.357 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.statistics: (nil) 2023-07-20T01:20:41.358 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.use_fsync: 0 2023-07-20T01:20:41.358 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.max_log_file_size: 0 2023-07-20T01:20:41.358 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-20T01:20:41.358 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-20T01:20:41.358 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.keep_log_file_num: 1000 2023-07-20T01:20:41.358 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.recycle_log_file_num: 0 2023-07-20T01:20:41.358 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.allow_fallocate: 1 2023-07-20T01:20:41.359 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.allow_mmap_reads: 0 2023-07-20T01:20:41.359 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.allow_mmap_writes: 0 2023-07-20T01:20:41.359 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.use_direct_reads: 0 2023-07-20T01:20:41.359 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-20T01:20:41.359 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.create_missing_column_families: 0 2023-07-20T01:20:41.359 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.db_log_dir: 2023-07-20T01:20:41.359 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.wal_dir: 2023-07-20T01:20:41.359 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-20T01:20:41.360 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-20T01:20:41.360 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-20T01:20:41.360 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-20T01:20:41.360 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-20T01:20:41.360 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-20T01:20:41.360 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.advise_random_on_open: 1 2023-07-20T01:20:41.360 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.db_write_buffer_size: 0 2023-07-20T01:20:41.361 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.write_buffer_manager: 0x55b938cc83c0 2023-07-20T01:20:41.361 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-20T01:20:41.361 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-20T01:20:41.361 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-20T01:20:41.361 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.rate_limiter: (nil) 2023-07-20T01:20:41.361 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-20T01:20:41.361 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.wal_recovery_mode: 2 2023-07-20T01:20:41.361 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.enable_thread_tracking: 0 2023-07-20T01:20:41.362 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.enable_pipelined_write: 0 2023-07-20T01:20:41.362 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.unordered_write: 0 2023-07-20T01:20:41.362 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-20T01:20:41.362 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-20T01:20:41.362 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-20T01:20:41.362 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-20T01:20:41.362 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.row_cache: None 2023-07-20T01:20:41.362 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.wal_filter: None 2023-07-20T01:20:41.363 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-20T01:20:41.363 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.allow_ingest_behind: 0 2023-07-20T01:20:41.363 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.two_write_queues: 0 2023-07-20T01:20:41.363 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.manual_wal_flush: 0 2023-07-20T01:20:41.363 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.wal_compression: 0 2023-07-20T01:20:41.363 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.atomic_flush: 0 2023-07-20T01:20:41.363 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-20T01:20:41.364 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-20T01:20:41.364 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-20T01:20:41.364 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.log_readahead_size: 0 2023-07-20T01:20:41.364 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-20T01:20:41.364 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.best_efforts_recovery: 0 2023-07-20T01:20:41.364 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-20T01:20:41.364 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-20T01:20:41.365 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.allow_data_in_errors: 0 2023-07-20T01:20:41.365 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.db_host_id: __hostname__ 2023-07-20T01:20:41.365 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-20T01:20:41.365 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.max_background_jobs: 2 2023-07-20T01:20:41.365 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.max_background_compactions: -1 2023-07-20T01:20:41.365 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.max_subcompactions: 1 2023-07-20T01:20:41.365 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-20T01:20:41.366 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-20T01:20:41.366 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-20T01:20:41.366 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.max_total_wal_size: 0 2023-07-20T01:20:41.366 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-20T01:20:41.366 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-20T01:20:41.366 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-20T01:20:41.366 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-20T01:20:41.366 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.max_open_files: -1 2023-07-20T01:20:41.367 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.bytes_per_sync: 0 2023-07-20T01:20:41.367 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-20T01:20:41.367 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-20T01:20:41.367 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.compaction_readahead_size: 0 2023-07-20T01:20:41.367 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.max_background_flushes: -1 2023-07-20T01:20:41.367 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Compression algorithms supported: 2023-07-20T01:20:41.368 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-20T01:20:41.368 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: kZSTD supported: 0 2023-07-20T01:20:41.368 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: kXpressCompression supported: 0 2023-07-20T01:20:41.368 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: kLZ4HCCompression supported: 1 2023-07-20T01:20:41.368 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: kZlibCompression supported: 1 2023-07-20T01:20:41.368 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: kSnappyCompression supported: 1 2023-07-20T01:20:41.369 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: kLZ4Compression supported: 1 2023-07-20T01:20:41.369 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: kBZip2Compression supported: 0 2023-07-20T01:20:41.369 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-20T01:20:41.369 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-20T01:20:41.369 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2023-07-20T01:20:41.370 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-20T01:20:41.370 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-20T01:20:41.370 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.merge_operator: 2023-07-20T01:20:41.370 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.compaction_filter: None 2023-07-20T01:20:41.370 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.compaction_filter_factory: None 2023-07-20T01:20:41.370 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.sst_partitioner_factory: None 2023-07-20T01:20:41.370 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-20T01:20:41.370 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-20T01:20:41.371 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b938da73c0) 2023-07-20T01:20:41.371 INFO:journalctl@ceph.mon.a.smithi060.stdout: cache_index_and_filter_blocks: 1 2023-07-20T01:20:41.371 INFO:journalctl@ceph.mon.a.smithi060.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-20T01:20:41.371 INFO:journalctl@ceph.mon.a.smithi060.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-20T01:20:41.371 INFO:journalctl@ceph.mon.a.smithi060.stdout: pin_top_level_index_and_filter: 1 2023-07-20T01:20:41.371 INFO:journalctl@ceph.mon.a.smithi060.stdout: index_type: 0 2023-07-20T01:20:41.371 INFO:journalctl@ceph.mon.a.smithi060.stdout: data_block_index_type: 0 2023-07-20T01:20:41.372 INFO:journalctl@ceph.mon.a.smithi060.stdout: index_shortening: 1 2023-07-20T01:20:41.372 INFO:journalctl@ceph.mon.a.smithi060.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-20T01:20:41.372 INFO:journalctl@ceph.mon.a.smithi060.stdout: checksum: 4 2023-07-20T01:20:41.372 INFO:journalctl@ceph.mon.a.smithi060.stdout: no_block_cache: 0 2023-07-20T01:20:41.372 INFO:journalctl@ceph.mon.a.smithi060.stdout: block_cache: 0x55b938d671f0 2023-07-20T01:20:41.372 INFO:journalctl@ceph.mon.a.smithi060.stdout: block_cache_name: BinnedLRUCache 2023-07-20T01:20:41.372 INFO:journalctl@ceph.mon.a.smithi060.stdout: block_cache_options: 2023-07-20T01:20:41.373 INFO:journalctl@ceph.mon.a.smithi060.stdout: capacity : 536870912 2023-07-20T01:20:41.373 INFO:journalctl@ceph.mon.a.smithi060.stdout: num_shard_bits : 4 2023-07-20T01:20:41.373 INFO:journalctl@ceph.mon.a.smithi060.stdout: strict_capacity_limit : 0 2023-07-20T01:20:41.373 INFO:journalctl@ceph.mon.a.smithi060.stdout: high_pri_pool_ratio: 0.000 2023-07-20T01:20:41.373 INFO:journalctl@ceph.mon.a.smithi060.stdout: block_cache_compressed: (nil) 2023-07-20T01:20:41.373 INFO:journalctl@ceph.mon.a.smithi060.stdout: persistent_cache: (nil) 2023-07-20T01:20:41.373 INFO:journalctl@ceph.mon.a.smithi060.stdout: block_size: 4096 2023-07-20T01:20:41.373 INFO:journalctl@ceph.mon.a.smithi060.stdout: block_size_deviation: 10 2023-07-20T01:20:41.374 INFO:journalctl@ceph.mon.a.smithi060.stdout: block_restart_interval: 16 2023-07-20T01:20:41.374 INFO:journalctl@ceph.mon.a.smithi060.stdout: index_block_restart_interval: 1 2023-07-20T01:20:41.374 INFO:journalctl@ceph.mon.a.smithi060.stdout: metadata_block_size: 4096 2023-07-20T01:20:41.374 INFO:journalctl@ceph.mon.a.smithi060.stdout: partition_filters: 0 2023-07-20T01:20:41.374 INFO:journalctl@ceph.mon.a.smithi060.stdout: use_delta_encoding: 1 2023-07-20T01:20:41.374 INFO:journalctl@ceph.mon.a.smithi060.stdout: filter_policy: bloomfilter 2023-07-20T01:20:41.374 INFO:journalctl@ceph.mon.a.smithi060.stdout: whole_key_filtering: 1 2023-07-20T01:20:41.374 INFO:journalctl@ceph.mon.a.smithi060.stdout: verify_compression: 0 2023-07-20T01:20:41.375 INFO:journalctl@ceph.mon.a.smithi060.stdout: read_amp_bytes_per_bit: 0 2023-07-20T01:20:41.375 INFO:journalctl@ceph.mon.a.smithi060.stdout: format_version: 5 2023-07-20T01:20:41.375 INFO:journalctl@ceph.mon.a.smithi060.stdout: enable_index_compression: 1 2023-07-20T01:20:41.375 INFO:journalctl@ceph.mon.a.smithi060.stdout: block_align: 0 2023-07-20T01:20:41.375 INFO:journalctl@ceph.mon.a.smithi060.stdout: max_auto_readahead_size: 262144 2023-07-20T01:20:41.375 INFO:journalctl@ceph.mon.a.smithi060.stdout: prepopulate_block_cache: 0 2023-07-20T01:20:41.375 INFO:journalctl@ceph.mon.a.smithi060.stdout: initial_auto_readahead_size: 8192 2023-07-20T01:20:41.375 INFO:journalctl@ceph.mon.a.smithi060.stdout: num_file_reads_for_auto_readahead: 2 2023-07-20T01:20:41.376 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.write_buffer_size: 33554432 2023-07-20T01:20:41.376 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.max_write_buffer_number: 2 2023-07-20T01:20:41.376 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.compression: NoCompression 2023-07-20T01:20:41.376 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.bottommost_compression: Disabled 2023-07-20T01:20:41.376 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.prefix_extractor: nullptr 2023-07-20T01:20:41.376 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-20T01:20:41.376 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.num_levels: 7 2023-07-20T01:20:41.377 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-20T01:20:41.377 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-20T01:20:41.377 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-20T01:20:41.377 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-20T01:20:41.377 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-20T01:20:41.377 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-20T01:20:41.377 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-20T01:20:41.377 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-20T01:20:41.378 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-20T01:20:41.378 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-20T01:20:41.378 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-20T01:20:41.378 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-20T01:20:41.378 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-20T01:20:41.378 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.compression_opts.level: 32767 2023-07-20T01:20:41.378 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.compression_opts.strategy: 0 2023-07-20T01:20:41.379 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-20T01:20:41.379 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-20T01:20:41.379 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-20T01:20:41.379 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-20T01:20:41.379 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.compression_opts.enabled: false 2023-07-20T01:20:41.379 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-20T01:20:41.379 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-20T01:20:41.379 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-20T01:20:41.380 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-20T01:20:41.380 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.target_file_size_base: 67108864 2023-07-20T01:20:41.380 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-20T01:20:41.380 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-20T01:20:41.380 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-20T01:20:41.380 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-20T01:20:41.380 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-20T01:20:41.380 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-20T01:20:41.381 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-20T01:20:41.381 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-20T01:20:41.381 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-20T01:20:41.381 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-20T01:20:41.381 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-20T01:20:41.381 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-20T01:20:41.381 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-20T01:20:41.382 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-20T01:20:41.382 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.arena_block_size: 1048576 2023-07-20T01:20:41.382 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-20T01:20:41.382 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-20T01:20:41.382 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.disable_auto_compactions: 0 2023-07-20T01:20:41.382 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-20T01:20:41.382 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-20T01:20:41.383 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-20T01:20:41.383 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-20T01:20:41.383 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-20T01:20:41.384 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-20T01:20:41.384 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-20T01:20:41.384 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-20T01:20:41.384 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-20T01:20:41.384 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-20T01:20:41.384 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.table_properties_collectors: 2023-07-20T01:20:41.385 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.inplace_update_support: 0 2023-07-20T01:20:41.385 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-20T01:20:41.385 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-20T01:20:41.385 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-20T01:20:41.385 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-20T01:20:41.385 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.bloom_locality: 0 2023-07-20T01:20:41.385 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.max_successive_merges: 0 2023-07-20T01:20:41.385 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-20T01:20:41.386 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.paranoid_file_checks: 0 2023-07-20T01:20:41.386 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.force_consistency_checks: 1 2023-07-20T01:20:41.386 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.report_bg_io_stats: 0 2023-07-20T01:20:41.386 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.ttl: 2592000 2023-07-20T01:20:41.386 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-20T01:20:41.386 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-20T01:20:41.386 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-20T01:20:41.386 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.enable_blob_files: false 2023-07-20T01:20:41.387 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.min_blob_size: 0 2023-07-20T01:20:41.387 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.blob_file_size: 268435456 2023-07-20T01:20:41.387 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-20T01:20:41.387 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-20T01:20:41.387 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-20T01:20:41.387 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-20T01:20:41.387 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-20T01:20:41.388 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.blob_file_starting_level: 0 2023-07-20T01:20:41.388 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-20T01:20:41.388 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2023-07-20T01:20:41.388 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2023-07-20T01:20:41.388 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7671d005-1a8d-4d09-b245-40eac824a76e 2023-07-20T01:20:41.388 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689816041299309, "job": 1, "event": "recovery_started", "wal_files": [9]} 2023-07-20T01:20:41.388 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2023-07-20T01:20:41.389 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689816041300292, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 94368, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 316, "table_properties": {"data_size": 92372, "index_size": 278, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13755, "raw_average_key_size": 50, "raw_value_size": 84861, "raw_average_value_size": 308, "num_data_blocks": 11, "num_entries": 275, "num_filter_entries": 275, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689816041, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7671d005-1a8d-4d09-b245-40eac824a76e", "db_session_id": "6W3497VOHB4T98ZDWLCP", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2023-07-20T01:20:41.389 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689816041300375, "job": 1, "event": "recovery_finished"} 2023-07-20T01:20:41.389 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2023-07-20T01:20:41.389 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-20T01:20:41.389 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55b938e00000 2023-07-20T01:20:41.389 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: DB pointer 0x55b938dec000 2023-07-20T01:20:41.389 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-20T01:20:41.390 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-20T01:20:41.390 INFO:journalctl@ceph.mon.a.smithi060.stdout: ** DB Stats ** 2023-07-20T01:20:41.390 INFO:journalctl@ceph.mon.a.smithi060.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T01:20:41.390 INFO:journalctl@ceph.mon.a.smithi060.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-20T01:20:41.390 INFO:journalctl@ceph.mon.a.smithi060.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T01:20:41.390 INFO:journalctl@ceph.mon.a.smithi060.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T01:20:41.390 INFO:journalctl@ceph.mon.a.smithi060.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-20T01:20:41.390 INFO:journalctl@ceph.mon.a.smithi060.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T01:20:41.391 INFO:journalctl@ceph.mon.a.smithi060.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T01:20:41.391 INFO:journalctl@ceph.mon.a.smithi060.stdout: 2023-07-20T01:20:41.391 INFO:journalctl@ceph.mon.a.smithi060.stdout: ** Compaction Stats [default] ** 2023-07-20T01:20:41.391 INFO:journalctl@ceph.mon.a.smithi060.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T01:20:41.391 INFO:journalctl@ceph.mon.a.smithi060.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-20T01:20:41.391 INFO:journalctl@ceph.mon.a.smithi060.stdout: L0 2/0 93.99 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T01:20:41.391 INFO:journalctl@ceph.mon.a.smithi060.stdout: Sum 2/0 93.99 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T01:20:41.391 INFO:journalctl@ceph.mon.a.smithi060.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T01:20:41.392 INFO:journalctl@ceph.mon.a.smithi060.stdout: 2023-07-20T01:20:41.392 INFO:journalctl@ceph.mon.a.smithi060.stdout: ** Compaction Stats [default] ** 2023-07-20T01:20:41.392 INFO:journalctl@ceph.mon.a.smithi060.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T01:20:41.392 INFO:journalctl@ceph.mon.a.smithi060.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-20T01:20:41.392 INFO:journalctl@ceph.mon.a.smithi060.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 159.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-20T01:20:41.392 INFO:journalctl@ceph.mon.a.smithi060.stdout: 2023-07-20T01:20:41.392 INFO:journalctl@ceph.mon.a.smithi060.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-20T01:20:41.393 INFO:journalctl@ceph.mon.a.smithi060.stdout: 2023-07-20T01:20:41.393 INFO:journalctl@ceph.mon.a.smithi060.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T01:20:41.393 INFO:journalctl@ceph.mon.a.smithi060.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-20T01:20:41.393 INFO:journalctl@ceph.mon.a.smithi060.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-20T01:20:41.393 INFO:journalctl@ceph.mon.a.smithi060.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-20T01:20:41.393 INFO:journalctl@ceph.mon.a.smithi060.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-20T01:20:41.393 INFO:journalctl@ceph.mon.a.smithi060.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-20T01:20:41.394 INFO:journalctl@ceph.mon.a.smithi060.stdout: Cumulative compaction: 0.00 GB write, 29.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T01:20:41.394 INFO:journalctl@ceph.mon.a.smithi060.stdout: Interval compaction: 0.00 GB write, 29.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T01:20:41.394 INFO:journalctl@ceph.mon.a.smithi060.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-20T01:20:41.394 INFO:journalctl@ceph.mon.a.smithi060.stdout: Block cache BinnedLRUCache@0x55b938d671f0#8 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-07-20T01:20:41.394 INFO:journalctl@ceph.mon.a.smithi060.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2023-07-20T01:20:41.394 INFO:journalctl@ceph.mon.a.smithi060.stdout: 2023-07-20T01:20:41.394 INFO:journalctl@ceph.mon.a.smithi060.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-20T01:20:41.394 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: starting mon.a rank 0 at public addrs [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] at bind addrs [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 7daf031e-269b-11ee-9b34-001a4aab830c 2023-07-20T01:20:41.395 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: mon.a@-1(???) e1 preinit fsid 7daf031e-269b-11ee-9b34-001a4aab830c 2023-07-20T01:20:41.395 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: mon.a@-1(???).mds e1 new map 2023-07-20T01:20:41.395 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: mon.a@-1(???).mds e1 print_map 2023-07-20T01:20:41.395 INFO:journalctl@ceph.mon.a.smithi060.stdout: e1 2023-07-20T01:20:41.395 INFO:journalctl@ceph.mon.a.smithi060.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-20T01:20:41.395 INFO:journalctl@ceph.mon.a.smithi060.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-20T01:20:41.395 INFO:journalctl@ceph.mon.a.smithi060.stdout: legacy client fscid: -1 2023-07-20T01:20:41.395 INFO:journalctl@ceph.mon.a.smithi060.stdout: 2023-07-20T01:20:41.396 INFO:journalctl@ceph.mon.a.smithi060.stdout: No filesystems configured 2023-07-20T01:20:41.396 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-20T01:20:41.396 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T01:20:41.396 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T01:20:41.396 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T01:20:41.396 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-07-20T01:20:41.396 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 systemd[1]: Started Ceph mon.a for 7daf031e-269b-11ee-9b34-001a4aab830c. 2023-07-20T01:20:41.397 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-20T01:20:41.397 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: monmap e1: 1 mons at {a=[v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0]} removed_ranks: {} 2023-07-20T01:20:41.397 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: fsmap 2023-07-20T01:20:41.397 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: osdmap e1: 0 total, 0 up, 0 in 2023-07-20T01:20:41.397 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:41 smithi060 ceph-mon[96524]: mgrmap e1: no daemons active 2023-07-20T01:20:42.356 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.868+0000 7f7805a64700 1 Processor -- start 2023-07-20T01:20:42.356 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.868+0000 7f7805a64700 1 -- start start 2023-07-20T01:20:42.357 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.869+0000 7f7805a64700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7800106a70 0x7f7800108e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:20:42.357 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.869+0000 7f7805a64700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7800073ec0 con 0x7f7800106a70 2023-07-20T01:20:42.357 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.869+0000 7f77ff7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7800106a70 0x7f7800108e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:20:42.358 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.869+0000 7f77ff7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7800106a70 0x7f7800108e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41786/0 (socket says 172.21.15.60:41786) 2023-07-20T01:20:42.358 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.869+0000 7f77ff7fe700 1 -- 172.21.15.60:0/1234448125 learned_addr learned my addr 172.21.15.60:0/1234448125 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:20:42.359 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.869+0000 7f77ff7fe700 1 -- 172.21.15.60:0/1234448125 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78001093a0 con 0x7f7800106a70 2023-07-20T01:20:42.359 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.870+0000 7f77ff7fe700 1 --2- 172.21.15.60:0/1234448125 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7800106a70 0x7f7800108e60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f77f00098a0 tx=0x7f77f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=399da764543b8ef3 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:20:42.359 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.870+0000 7f77fe7fc700 1 -- 172.21.15.60:0/1234448125 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f77f0004160 con 0x7f7800106a70 2023-07-20T01:20:42.359 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.870+0000 7f77fe7fc700 1 -- 172.21.15.60:0/1234448125 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f77f00042c0 con 0x7f7800106a70 2023-07-20T01:20:42.360 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.870+0000 7f77fe7fc700 1 -- 172.21.15.60:0/1234448125 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f77f0011940 con 0x7f7800106a70 2023-07-20T01:20:42.360 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.870+0000 7f7805a64700 1 -- 172.21.15.60:0/1234448125 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7800106a70 msgr2=0x7f7800108e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:20:42.360 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.870+0000 7f7805a64700 1 --2- 172.21.15.60:0/1234448125 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7800106a70 0x7f7800108e60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f77f00098a0 tx=0x7f77f0009bb0 comp rx=0 tx=0).stop 2023-07-20T01:20:42.361 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.870+0000 7f7805a64700 1 -- 172.21.15.60:0/1234448125 shutdown_connections 2023-07-20T01:20:42.361 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.870+0000 7f7805a64700 1 --2- 172.21.15.60:0/1234448125 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7800106a70 0x7f7800108e60 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:20:42.361 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.870+0000 7f7805a64700 1 -- 172.21.15.60:0/1234448125 >> 172.21.15.60:0/1234448125 conn(0x7f78001004a0 msgr2=0x7f78001028c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:20:42.361 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.870+0000 7f7805a64700 1 -- 172.21.15.60:0/1234448125 shutdown_connections 2023-07-20T01:20:42.361 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.870+0000 7f7805a64700 1 -- 172.21.15.60:0/1234448125 wait complete. 2023-07-20T01:20:42.362 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.871+0000 7f7805a64700 1 Processor -- start 2023-07-20T01:20:42.362 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.871+0000 7f7805a64700 1 -- start start 2023-07-20T01:20:42.362 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.871+0000 7f7805a64700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7800106a70 0x7f78001979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:20:42.362 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.871+0000 7f7805a64700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7800197f10 con 0x7f7800106a70 2023-07-20T01:20:42.362 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.872+0000 7f77ff7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7800106a70 0x7f78001979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:20:42.362 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.872+0000 7f77ff7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7800106a70 0x7f78001979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41788/0 (socket says 172.21.15.60:41788) 2023-07-20T01:20:42.362 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.872+0000 7f77ff7fe700 1 -- 172.21.15.60:0/3533051139 learned_addr learned my addr 172.21.15.60:0/3533051139 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:20:42.363 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.872+0000 7f77ff7fe700 1 -- 172.21.15.60:0/3533051139 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f0009550 con 0x7f7800106a70 2023-07-20T01:20:42.363 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.872+0000 7f77ff7fe700 1 --2- 172.21.15.60:0/3533051139 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7800106a70 0x7f78001979d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f77f0011dc0 tx=0x7f77f0011ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:20:42.363 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.872+0000 7f77fcff9700 1 -- 172.21.15.60:0/3533051139 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f77f00037d0 con 0x7f7800106a70 2023-07-20T01:20:42.363 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.872+0000 7f77fcff9700 1 -- 172.21.15.60:0/3533051139 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f77f001b440 con 0x7f7800106a70 2023-07-20T01:20:42.363 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.872+0000 7f77fcff9700 1 -- 172.21.15.60:0/3533051139 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f77f001a560 con 0x7f7800106a70 2023-07-20T01:20:42.363 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.872+0000 7f7805a64700 1 -- 172.21.15.60:0/3533051139 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7800198110 con 0x7f7800106a70 2023-07-20T01:20:42.363 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.872+0000 7f7805a64700 1 -- 172.21.15.60:0/3533051139 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7800198530 con 0x7f7800106a70 2023-07-20T01:20:42.364 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.872+0000 7f77fcff9700 1 -- 172.21.15.60:0/3533051139 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f77f001aad0 con 0x7f7800106a70 2023-07-20T01:20:42.364 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.872+0000 7f77fcff9700 1 -- 172.21.15.60:0/3533051139 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f77f00193f0 con 0x7f7800106a70 2023-07-20T01:20:42.364 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.873+0000 7f7805a64700 1 -- 172.21.15.60:0/3533051139 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7800191ee0 con 0x7f7800106a70 2023-07-20T01:20:42.364 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:41.876+0000 7f77fcff9700 1 -- 172.21.15.60:0/3533051139 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f77f0029060 con 0x7f7800106a70 2023-07-20T01:20:42.364 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:42.001+0000 7f7805a64700 1 -- 172.21.15.60:0/3533051139 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f780002ce00 con 0x7f7800106a70 2023-07-20T01:20:42.364 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:42.004+0000 7f77fcff9700 1 -- 172.21.15.60:0/3533051139 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f77f00330c0 con 0x7f7800106a70 2023-07-20T01:20:42.364 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:42.004+0000 7f7805a64700 1 -- 172.21.15.60:0/3533051139 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7800106a70 msgr2=0x7f78001979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:20:42.365 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:42.004+0000 7f7805a64700 1 --2- 172.21.15.60:0/3533051139 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7800106a70 0x7f78001979d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f77f0011dc0 tx=0x7f77f0011ea0 comp rx=0 tx=0).stop 2023-07-20T01:20:42.365 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:42.004+0000 7f7805a64700 1 -- 172.21.15.60:0/3533051139 shutdown_connections 2023-07-20T01:20:42.365 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:42.004+0000 7f7805a64700 1 --2- 172.21.15.60:0/3533051139 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7800106a70 0x7f78001979d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:20:42.365 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:42.004+0000 7f7805a64700 1 -- 172.21.15.60:0/3533051139 >> 172.21.15.60:0/3533051139 conn(0x7f78001004a0 msgr2=0x7f7800101110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:20:42.365 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:42.005+0000 7f7805a64700 1 -- 172.21.15.60:0/3533051139 shutdown_connections 2023-07-20T01:20:42.365 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:42.005+0000 7f7805a64700 1 -- 172.21.15.60:0/3533051139 wait complete. 2023-07-20T01:20:42.365 INFO:teuthology.orchestra.run.smithi060.stdout:Wrote config to /etc/ceph/ceph.conf 2023-07-20T01:20:42.391 INFO:teuthology.orchestra.run.smithi060.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-07-20T01:20:42.391 INFO:teuthology.orchestra.run.smithi060.stdout:Creating mgr... 2023-07-20T01:20:42.391 INFO:teuthology.orchestra.run.smithi060.stdout:Verifying port 9283 ... 2023-07-20T01:20:42.392 INFO:teuthology.orchestra.run.smithi060.stdout:Verifying port 8765 ... 2023-07-20T01:20:42.604 INFO:teuthology.orchestra.run.smithi060.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mgr.x 2023-07-20T01:20:42.605 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stderr Failed to reset failed state of unit ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mgr.x.service: Unit ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mgr.x.service not loaded. 2023-07-20T01:20:42.701 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-7daf031e-269b-11ee-9b34-001a4aab830c.target.wants/ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mgr.x.service → /etc/systemd/system/ceph-7daf031e-269b-11ee-9b34-001a4aab830c@.service. 2023-07-20T01:20:42.986 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:20:42 smithi060 systemd[1]: Starting Ceph mgr.x for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:20:43.250 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:43 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/3533051139' entity='client.admin' 2023-07-20T01:20:43.251 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:20:43 smithi060 podman[96754]: 2023-07-20T01:20:43.309 INFO:teuthology.orchestra.run.smithi060.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-20T01:20:43.309 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stdout disabled 2023-07-20T01:20:43.318 INFO:teuthology.orchestra.run.smithi060.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-20T01:20:43.318 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stdout inactive 2023-07-20T01:20:43.318 INFO:teuthology.orchestra.run.smithi060.stdout:firewalld.service is not enabled 2023-07-20T01:20:43.319 INFO:teuthology.orchestra.run.smithi060.stdout:Not possible to enable service . firewalld.service is not available 2023-07-20T01:20:43.327 INFO:teuthology.orchestra.run.smithi060.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-20T01:20:43.327 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stdout disabled 2023-07-20T01:20:43.335 INFO:teuthology.orchestra.run.smithi060.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-20T01:20:43.336 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stdout inactive 2023-07-20T01:20:43.336 INFO:teuthology.orchestra.run.smithi060.stdout:firewalld.service is not enabled 2023-07-20T01:20:43.336 INFO:teuthology.orchestra.run.smithi060.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2023-07-20T01:20:43.336 INFO:teuthology.orchestra.run.smithi060.stdout:Waiting for mgr to start... 2023-07-20T01:20:43.336 INFO:teuthology.orchestra.run.smithi060.stdout:Waiting for mgr... 2023-07-20T01:20:43.506 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:20:43 smithi060 bash[96754]: 4a99523bef7fdbf0e82b32e640e191da6b446687065021f2cebfcdc1eb07ffdd 2023-07-20T01:20:43.507 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:20:43 smithi060 systemd[1]: Started Ceph mgr.x for 7daf031e-269b-11ee-9b34-001a4aab830c. 2023-07-20T01:20:43.507 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:20:43 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:20:43.425+0000 7f3c9be43200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T01:20:43.507 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:20:43 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:20:43.523+0000 7f3c9be43200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T01:20:43.930 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:20:43 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:20:43.637+0000 7f3c9be43200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T01:20:43.930 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:20:43 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:20:43.750+0000 7f3c9be43200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T01:20:44.365 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:44 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/910232698' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-20T01:20:44.365 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:20:43 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:20:43.974+0000 7f3c9be43200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T01:20:44.365 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:20:44 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:20:44.381+0000 7f3c9be43200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T01:20:44.488 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 2023-07-20T01:20:44.489 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout { 2023-07-20T01:20:44.489 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "fsid": "7daf031e-269b-11ee-9b34-001a4aab830c", 2023-07-20T01:20:44.489 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "health": { 2023-07-20T01:20:44.489 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-20T01:20:44.489 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-20T01:20:44.489 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-20T01:20:44.489 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:20:44.490 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-20T01:20:44.490 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-20T01:20:44.490 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 0 2023-07-20T01:20:44.490 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout ], 2023-07-20T01:20:44.502 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-20T01:20:44.502 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "a" 2023-07-20T01:20:44.503 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout ], 2023-07-20T01:20:44.503 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2023-07-20T01:20:44.503 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-20T01:20:44.504 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T01:20:44.504 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-20T01:20:44.504 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-20T01:20:44.504 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:20:44.504 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-20T01:20:44.504 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T01:20:44.504 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-20T01:20:44.505 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-20T01:20:44.505 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-20T01:20:44.505 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-20T01:20:44.505 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-20T01:20:44.505 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-20T01:20:44.505 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:20:44.505 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-20T01:20:44.506 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-20T01:20:44.506 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-20T01:20:44.506 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-20T01:20:44.506 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-20T01:20:44.506 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-20T01:20:44.506 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-20T01:20:44.506 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-20T01:20:44.506 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-20T01:20:44.507 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:20:44.507 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-20T01:20:44.507 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T01:20:44.507 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-20T01:20:44.507 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-20T01:20:44.507 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:20:44.507 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-20T01:20:44.507 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-20T01:20:44.508 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-20T01:20:44.508 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-20T01:20:44.508 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-20T01:20:44.508 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-20T01:20:44.508 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "restful" 2023-07-20T01:20:44.508 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout ], 2023-07-20T01:20:44.508 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T01:20:44.509 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:20:44.509 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-20T01:20:44.509 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T01:20:44.509 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "modified": "2023-07-20T01:20:36.943821+0000", 2023-07-20T01:20:44.509 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T01:20:44.509 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:20:44.509 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-20T01:20:44.509 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout } 2023-07-20T01:20:44.510 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.905+0000 7fb74013c700 1 Processor -- start 2023-07-20T01:20:44.510 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.906+0000 7fb74013c700 1 -- start start 2023-07-20T01:20:44.510 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.906+0000 7fb74013c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb738104890 0x7fb738104cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:20:44.510 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.906+0000 7fb74013c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb738073e80 con 0x7fb738104890 2023-07-20T01:20:44.510 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.906+0000 7fb73ded8700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb738104890 0x7fb738104cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:20:44.510 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.906+0000 7fb73ded8700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb738104890 0x7fb738104cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41794/0 (socket says 172.21.15.60:41794) 2023-07-20T01:20:44.510 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.906+0000 7fb73ded8700 1 -- 172.21.15.60:0/444886189 learned_addr learned my addr 172.21.15.60:0/444886189 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:20:44.510 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.907+0000 7fb73ded8700 1 -- 172.21.15.60:0/444886189 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7381051f0 con 0x7fb738104890 2023-07-20T01:20:44.511 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.907+0000 7fb73ded8700 1 --2- 172.21.15.60:0/444886189 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb738104890 0x7fb738104cb0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb7340098a0 tx=0x7fb734009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9026df910fa63252 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:20:44.511 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.907+0000 7fb73ced6700 1 -- 172.21.15.60:0/444886189 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb734004160 con 0x7fb738104890 2023-07-20T01:20:44.511 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.907+0000 7fb73ced6700 1 -- 172.21.15.60:0/444886189 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fb7340042c0 con 0x7fb738104890 2023-07-20T01:20:44.511 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.907+0000 7fb73ced6700 1 -- 172.21.15.60:0/444886189 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb734011940 con 0x7fb738104890 2023-07-20T01:20:44.511 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.908+0000 7fb74013c700 1 -- 172.21.15.60:0/444886189 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb738104890 msgr2=0x7fb738104cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:20:44.512 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.908+0000 7fb74013c700 1 --2- 172.21.15.60:0/444886189 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb738104890 0x7fb738104cb0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb7340098a0 tx=0x7fb734009bb0 comp rx=0 tx=0).stop 2023-07-20T01:20:44.512 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.908+0000 7fb74013c700 1 -- 172.21.15.60:0/444886189 shutdown_connections 2023-07-20T01:20:44.512 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.908+0000 7fb74013c700 1 --2- 172.21.15.60:0/444886189 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb738104890 0x7fb738104cb0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:20:44.512 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.908+0000 7fb74013c700 1 -- 172.21.15.60:0/444886189 >> 172.21.15.60:0/444886189 conn(0x7fb738100490 msgr2=0x7fb7381028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:20:44.512 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.908+0000 7fb74013c700 1 -- 172.21.15.60:0/444886189 shutdown_connections 2023-07-20T01:20:44.512 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.908+0000 7fb74013c700 1 -- 172.21.15.60:0/444886189 wait complete. 2023-07-20T01:20:44.512 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.909+0000 7fb74013c700 1 Processor -- start 2023-07-20T01:20:44.512 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.909+0000 7fb74013c700 1 -- start start 2023-07-20T01:20:44.513 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.909+0000 7fb74013c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb738104890 0x7fb73819bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:20:44.513 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.909+0000 7fb74013c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73819c2f0 con 0x7fb738104890 2023-07-20T01:20:44.513 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.909+0000 7fb73ded8700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb738104890 0x7fb73819bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:20:44.513 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.909+0000 7fb73ded8700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb738104890 0x7fb73819bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41796/0 (socket says 172.21.15.60:41796) 2023-07-20T01:20:44.513 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.909+0000 7fb73ded8700 1 -- 172.21.15.60:0/910232698 learned_addr learned my addr 172.21.15.60:0/910232698 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:20:44.513 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.909+0000 7fb73ded8700 1 -- 172.21.15.60:0/910232698 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb734009550 con 0x7fb738104890 2023-07-20T01:20:44.513 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.909+0000 7fb73ded8700 1 --2- 172.21.15.60:0/910232698 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb738104890 0x7fb73819bdb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb734006af0 tx=0x7fb734011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:20:44.514 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.910+0000 7fb72effd700 1 -- 172.21.15.60:0/910232698 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb734003bf0 con 0x7fb738104890 2023-07-20T01:20:44.514 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.910+0000 7fb72effd700 1 -- 172.21.15.60:0/910232698 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fb73401b8b0 con 0x7fb738104890 2023-07-20T01:20:44.514 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.910+0000 7fb72effd700 1 -- 172.21.15.60:0/910232698 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb734019880 con 0x7fb738104890 2023-07-20T01:20:44.514 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.910+0000 7fb74013c700 1 -- 172.21.15.60:0/910232698 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb73819c4f0 con 0x7fb738104890 2023-07-20T01:20:44.514 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.910+0000 7fb74013c700 1 -- 172.21.15.60:0/910232698 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb73819c910 con 0x7fb738104890 2023-07-20T01:20:44.514 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.910+0000 7fb72effd700 1 -- 172.21.15.60:0/910232698 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb7340233f0 con 0x7fb738104890 2023-07-20T01:20:44.514 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.910+0000 7fb72effd700 1 -- 172.21.15.60:0/910232698 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb73401a710 con 0x7fb738104890 2023-07-20T01:20:44.515 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.911+0000 7fb74013c700 1 -- 172.21.15.60:0/910232698 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb738195670 con 0x7fb738104890 2023-07-20T01:20:44.515 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:43.913+0000 7fb72effd700 1 -- 172.21.15.60:0/910232698 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fb734028060 con 0x7fb738104890 2023-07-20T01:20:44.515 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:44.057+0000 7fb74013c700 1 -- 172.21.15.60:0/910232698 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb73802c970 con 0x7fb738104890 2023-07-20T01:20:44.515 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:44.057+0000 7fb72effd700 1 -- 172.21.15.60:0/910232698 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fb73401bd70 con 0x7fb738104890 2023-07-20T01:20:44.515 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:44.058+0000 7fb74013c700 1 -- 172.21.15.60:0/910232698 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb738104890 msgr2=0x7fb73819bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:20:44.515 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:44.058+0000 7fb74013c700 1 --2- 172.21.15.60:0/910232698 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb738104890 0x7fb73819bdb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb734006af0 tx=0x7fb734011a30 comp rx=0 tx=0).stop 2023-07-20T01:20:44.515 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:44.058+0000 7fb74013c700 1 -- 172.21.15.60:0/910232698 shutdown_connections 2023-07-20T01:20:44.515 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:44.058+0000 7fb74013c700 1 --2- 172.21.15.60:0/910232698 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb738104890 0x7fb73819bdb0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:20:44.516 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:44.058+0000 7fb74013c700 1 -- 172.21.15.60:0/910232698 >> 172.21.15.60:0/910232698 conn(0x7fb738100490 msgr2=0x7fb738190450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:20:44.516 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:44.058+0000 7fb74013c700 1 -- 172.21.15.60:0/910232698 shutdown_connections 2023-07-20T01:20:44.516 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:44.059+0000 7fb74013c700 1 -- 172.21.15.60:0/910232698 wait complete. 2023-07-20T01:20:44.516 INFO:teuthology.orchestra.run.smithi060.stdout:mgr not available, waiting (1/15)... 2023-07-20T01:20:44.625 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:20:44 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:20:44.490+0000 7f3c9be43200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T01:20:44.929 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:20:44 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:20:44.641+0000 7f3c9be43200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T01:20:44.930 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:20:44 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:20:44.757+0000 7f3c9be43200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T01:20:45.679 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:20:45 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:20:45.267+0000 7f3c9be43200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T01:20:46.429 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:20:45 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:20:45.962+0000 7f3c9be43200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T01:20:46.430 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:20:46 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:20:46.153+0000 7f3c9be43200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T01:20:47.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:47 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/305903428' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-20T01:20:47.430 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:20:47 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:20:47.134+0000 7f3c9be43200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T01:20:47.430 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:20:47 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:20:47.247+0000 7f3c9be43200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T01:20:47.641 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 2023-07-20T01:20:47.641 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout { 2023-07-20T01:20:47.641 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "fsid": "7daf031e-269b-11ee-9b34-001a4aab830c", 2023-07-20T01:20:47.641 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "health": { 2023-07-20T01:20:47.642 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-20T01:20:47.642 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-20T01:20:47.642 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-20T01:20:47.642 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:20:47.642 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-20T01:20:47.642 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-20T01:20:47.642 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 0 2023-07-20T01:20:47.642 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout ], 2023-07-20T01:20:47.643 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-20T01:20:47.643 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "a" 2023-07-20T01:20:47.643 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout ], 2023-07-20T01:20:47.643 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2023-07-20T01:20:47.643 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-20T01:20:47.643 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T01:20:47.643 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-20T01:20:47.643 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-20T01:20:47.644 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:20:47.644 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-20T01:20:47.644 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T01:20:47.644 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-20T01:20:47.644 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-20T01:20:47.644 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-20T01:20:47.650 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-20T01:20:47.650 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-20T01:20:47.650 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-20T01:20:47.650 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:20:47.650 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-20T01:20:47.650 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-20T01:20:47.650 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-20T01:20:47.651 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-20T01:20:47.651 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-20T01:20:47.651 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-20T01:20:47.651 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-20T01:20:47.651 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-20T01:20:47.651 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-20T01:20:47.651 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:20:47.652 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-20T01:20:47.652 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T01:20:47.652 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-20T01:20:47.652 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-20T01:20:47.652 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:20:47.652 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-20T01:20:47.652 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-20T01:20:47.653 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-20T01:20:47.653 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-20T01:20:47.653 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-20T01:20:47.653 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-20T01:20:47.653 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "restful" 2023-07-20T01:20:47.653 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout ], 2023-07-20T01:20:47.653 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T01:20:47.653 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:20:47.654 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-20T01:20:47.654 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T01:20:47.654 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "modified": "2023-07-20T01:20:36.943821+0000", 2023-07-20T01:20:47.654 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T01:20:47.654 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:20:47.654 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-20T01:20:47.654 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout } 2023-07-20T01:20:47.654 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.047+0000 7f2090791700 1 Processor -- start 2023-07-20T01:20:47.655 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.047+0000 7f2090791700 1 -- start start 2023-07-20T01:20:47.655 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.047+0000 7f2090791700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2088106a20 0x7f2088108e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:20:47.655 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.047+0000 7f2090791700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2088073f00 con 0x7f2088106a20 2023-07-20T01:20:47.655 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.048+0000 7f208e52d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2088106a20 0x7f2088108e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:20:47.655 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.048+0000 7f208e52d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2088106a20 0x7f2088108e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41798/0 (socket says 172.21.15.60:41798) 2023-07-20T01:20:47.655 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.048+0000 7f208e52d700 1 -- 172.21.15.60:0/2281564740 learned_addr learned my addr 172.21.15.60:0/2281564740 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:20:47.655 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.048+0000 7f208e52d700 1 -- 172.21.15.60:0/2281564740 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2088109350 con 0x7f2088106a20 2023-07-20T01:20:47.655 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.048+0000 7f208e52d700 1 --2- 172.21.15.60:0/2281564740 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2088106a20 0x7f2088108e10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f20840098a0 tx=0x7f2084009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71c3cb6393ae7233 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:20:47.656 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.048+0000 7f208d52b700 1 -- 172.21.15.60:0/2281564740 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2084004160 con 0x7f2088106a20 2023-07-20T01:20:47.656 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.048+0000 7f208d52b700 1 -- 172.21.15.60:0/2281564740 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f20840042c0 con 0x7f2088106a20 2023-07-20T01:20:47.656 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.048+0000 7f208d52b700 1 -- 172.21.15.60:0/2281564740 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2084011940 con 0x7f2088106a20 2023-07-20T01:20:47.656 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.049+0000 7f2090791700 1 -- 172.21.15.60:0/2281564740 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2088106a20 msgr2=0x7f2088108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:20:47.656 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.049+0000 7f2090791700 1 --2- 172.21.15.60:0/2281564740 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2088106a20 0x7f2088108e10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f20840098a0 tx=0x7f2084009bb0 comp rx=0 tx=0).stop 2023-07-20T01:20:47.656 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.049+0000 7f2090791700 1 -- 172.21.15.60:0/2281564740 shutdown_connections 2023-07-20T01:20:47.656 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.049+0000 7f2090791700 1 --2- 172.21.15.60:0/2281564740 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2088106a20 0x7f2088108e10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:20:47.657 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.049+0000 7f2090791700 1 -- 172.21.15.60:0/2281564740 >> 172.21.15.60:0/2281564740 conn(0x7f2088100490 msgr2=0x7f20881028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:20:47.657 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.049+0000 7f2090791700 1 -- 172.21.15.60:0/2281564740 shutdown_connections 2023-07-20T01:20:47.657 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.049+0000 7f2090791700 1 -- 172.21.15.60:0/2281564740 wait complete. 2023-07-20T01:20:47.657 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.050+0000 7f2090791700 1 Processor -- start 2023-07-20T01:20:47.657 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.050+0000 7f2090791700 1 -- start start 2023-07-20T01:20:47.657 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.050+0000 7f2090791700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2088106a20 0x7f2088106380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:20:47.657 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.050+0000 7f2090791700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20881068c0 con 0x7f2088106a20 2023-07-20T01:20:47.657 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.050+0000 7f208e52d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2088106a20 0x7f2088106380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:20:47.658 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.050+0000 7f208e52d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2088106a20 0x7f2088106380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41800/0 (socket says 172.21.15.60:41800) 2023-07-20T01:20:47.658 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.050+0000 7f208e52d700 1 -- 172.21.15.60:0/305903428 learned_addr learned my addr 172.21.15.60:0/305903428 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:20:47.658 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.051+0000 7f208e52d700 1 -- 172.21.15.60:0/305903428 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2084009550 con 0x7f2088106a20 2023-07-20T01:20:47.658 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.051+0000 7f208e52d700 1 --2- 172.21.15.60:0/305903428 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2088106a20 0x7f2088106380 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2084006af0 tx=0x7f2084011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:20:47.658 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.051+0000 7f207f7fe700 1 -- 172.21.15.60:0/305903428 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2084003bf0 con 0x7f2088106a20 2023-07-20T01:20:47.658 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.051+0000 7f207f7fe700 1 -- 172.21.15.60:0/305903428 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f208401b8b0 con 0x7f2088106a20 2023-07-20T01:20:47.658 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.051+0000 7f207f7fe700 1 -- 172.21.15.60:0/305903428 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2084019880 con 0x7f2088106a20 2023-07-20T01:20:47.659 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.051+0000 7f2090791700 1 -- 172.21.15.60:0/305903428 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20881049d0 con 0x7f2088106a20 2023-07-20T01:20:47.659 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.051+0000 7f2090791700 1 -- 172.21.15.60:0/305903428 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2088104df0 con 0x7f2088106a20 2023-07-20T01:20:47.659 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.051+0000 7f207f7fe700 1 -- 172.21.15.60:0/305903428 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f20840233f0 con 0x7f2088106a20 2023-07-20T01:20:47.659 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.052+0000 7f207f7fe700 1 -- 172.21.15.60:0/305903428 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f208401a8b0 con 0x7f2088106a20 2023-07-20T01:20:47.659 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.053+0000 7f2090791700 1 -- 172.21.15.60:0/305903428 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f208810a0f0 con 0x7f2088106a20 2023-07-20T01:20:47.659 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.055+0000 7f207f7fe700 1 -- 172.21.15.60:0/305903428 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f2084042b10 con 0x7f2088106a20 2023-07-20T01:20:47.659 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.187+0000 7f2090791700 1 -- 172.21.15.60:0/305903428 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f2088105450 con 0x7f2088106a20 2023-07-20T01:20:47.659 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.187+0000 7f207f7fe700 1 -- 172.21.15.60:0/305903428 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f2084031070 con 0x7f2088106a20 2023-07-20T01:20:47.660 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.188+0000 7f2090791700 1 -- 172.21.15.60:0/305903428 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2088106a20 msgr2=0x7f2088106380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:20:47.660 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.188+0000 7f2090791700 1 --2- 172.21.15.60:0/305903428 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2088106a20 0x7f2088106380 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2084006af0 tx=0x7f2084011a30 comp rx=0 tx=0).stop 2023-07-20T01:20:47.660 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.188+0000 7f2090791700 1 -- 172.21.15.60:0/305903428 shutdown_connections 2023-07-20T01:20:47.660 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.188+0000 7f2090791700 1 --2- 172.21.15.60:0/305903428 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2088106a20 0x7f2088106380 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:20:47.660 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.188+0000 7f2090791700 1 -- 172.21.15.60:0/305903428 >> 172.21.15.60:0/305903428 conn(0x7f2088100490 msgr2=0x7f2088101250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:20:47.660 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.188+0000 7f2090791700 1 -- 172.21.15.60:0/305903428 shutdown_connections 2023-07-20T01:20:47.660 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:47.188+0000 7f2090791700 1 -- 172.21.15.60:0/305903428 wait complete. 2023-07-20T01:20:47.660 INFO:teuthology.orchestra.run.smithi060.stdout:mgr not available, waiting (2/15)... 2023-07-20T01:20:47.813 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:20:47 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:20:47.479+0000 7f3c9be43200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T01:20:48.068 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:20:47 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:20:47.829+0000 7f3c9be43200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T01:20:48.429 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:20:48 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:20:48.084+0000 7f3c9be43200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T01:20:48.430 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:20:48 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:20:48.266+0000 7f3c9be43200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T01:20:48.929 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:20:48 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:20:48.591+0000 7f3c9be43200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T01:20:49.679 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:20:49 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:20:49.415+0000 7f3c9be43200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T01:20:49.680 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:20:49 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:20:49.575+0000 7f3c9be43200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T01:20:50.179 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:20:49 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:20:49.827+0000 7f3c9be43200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T01:20:50.180 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:20:49 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:20:49.968+0000 7f3c9be43200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T01:20:50.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:50 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/1442923720' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-20T01:20:50.680 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:20:50 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:20:50.243+0000 7f3c9be43200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T01:20:50.691 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 2023-07-20T01:20:50.691 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout { 2023-07-20T01:20:50.692 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "fsid": "7daf031e-269b-11ee-9b34-001a4aab830c", 2023-07-20T01:20:50.692 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "health": { 2023-07-20T01:20:50.692 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-20T01:20:50.692 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-20T01:20:50.692 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-20T01:20:50.692 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:20:50.693 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-20T01:20:50.693 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-20T01:20:50.693 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 0 2023-07-20T01:20:50.693 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout ], 2023-07-20T01:20:50.693 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-20T01:20:50.693 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "a" 2023-07-20T01:20:50.693 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout ], 2023-07-20T01:20:50.694 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2023-07-20T01:20:50.694 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-20T01:20:50.694 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T01:20:50.694 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-20T01:20:50.694 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-20T01:20:50.694 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:20:50.694 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-20T01:20:50.694 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T01:20:50.695 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-20T01:20:50.695 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-20T01:20:50.695 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-20T01:20:50.701 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-20T01:20:50.701 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-20T01:20:50.701 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-20T01:20:50.701 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:20:50.702 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-20T01:20:50.702 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-20T01:20:50.702 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-20T01:20:50.702 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-20T01:20:50.702 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-20T01:20:50.702 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-20T01:20:50.702 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-20T01:20:50.703 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-20T01:20:50.703 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-20T01:20:50.703 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:20:50.703 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-20T01:20:50.703 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T01:20:50.703 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-20T01:20:50.703 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-20T01:20:50.704 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:20:50.704 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-20T01:20:50.704 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-20T01:20:50.704 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-20T01:20:50.704 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-20T01:20:50.704 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-20T01:20:50.704 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-20T01:20:50.705 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "restful" 2023-07-20T01:20:50.705 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout ], 2023-07-20T01:20:50.705 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T01:20:50.705 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:20:50.705 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-20T01:20:50.705 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T01:20:50.705 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "modified": "2023-07-20T01:20:36.943821+0000", 2023-07-20T01:20:50.706 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T01:20:50.706 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:20:50.708 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-20T01:20:50.708 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout } 2023-07-20T01:20:50.708 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.171+0000 7f6875945700 1 Processor -- start 2023-07-20T01:20:50.708 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.171+0000 7f6875945700 1 -- start start 2023-07-20T01:20:50.708 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.172+0000 7f6875945700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6870106a20 0x7f6870108e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:20:50.709 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.172+0000 7f6875945700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6870073f00 con 0x7f6870106a20 2023-07-20T01:20:50.709 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.172+0000 7f686effd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6870106a20 0x7f6870108e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:20:50.709 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.172+0000 7f686effd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6870106a20 0x7f6870108e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41802/0 (socket says 172.21.15.60:41802) 2023-07-20T01:20:50.709 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.172+0000 7f686effd700 1 -- 172.21.15.60:0/24916401 learned_addr learned my addr 172.21.15.60:0/24916401 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:20:50.709 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.172+0000 7f686effd700 1 -- 172.21.15.60:0/24916401 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6870109350 con 0x7f6870106a20 2023-07-20T01:20:50.709 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.172+0000 7f686effd700 1 --2- 172.21.15.60:0/24916401 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6870106a20 0x7f6870108e10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f68600098a0 tx=0x7f6860009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=410e46de9c65a125 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:20:50.710 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.172+0000 7f686dffb700 1 -- 172.21.15.60:0/24916401 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6860004160 con 0x7f6870106a20 2023-07-20T01:20:50.710 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.172+0000 7f686dffb700 1 -- 172.21.15.60:0/24916401 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f68600042c0 con 0x7f6870106a20 2023-07-20T01:20:50.710 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.172+0000 7f686dffb700 1 -- 172.21.15.60:0/24916401 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6860011940 con 0x7f6870106a20 2023-07-20T01:20:50.710 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.173+0000 7f6875945700 1 -- 172.21.15.60:0/24916401 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6870106a20 msgr2=0x7f6870108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:20:50.710 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.173+0000 7f6875945700 1 --2- 172.21.15.60:0/24916401 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6870106a20 0x7f6870108e10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f68600098a0 tx=0x7f6860009bb0 comp rx=0 tx=0).stop 2023-07-20T01:20:50.710 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.173+0000 7f6875945700 1 -- 172.21.15.60:0/24916401 shutdown_connections 2023-07-20T01:20:50.710 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.173+0000 7f6875945700 1 --2- 172.21.15.60:0/24916401 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6870106a20 0x7f6870108e10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:20:50.711 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.173+0000 7f6875945700 1 -- 172.21.15.60:0/24916401 >> 172.21.15.60:0/24916401 conn(0x7f6870100490 msgr2=0x7f68701028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:20:50.711 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.173+0000 7f6875945700 1 -- 172.21.15.60:0/24916401 shutdown_connections 2023-07-20T01:20:50.711 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.173+0000 7f6875945700 1 -- 172.21.15.60:0/24916401 wait complete. 2023-07-20T01:20:50.711 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.174+0000 7f6875945700 1 Processor -- start 2023-07-20T01:20:50.711 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.174+0000 7f6875945700 1 -- start start 2023-07-20T01:20:50.711 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.174+0000 7f6875945700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6870106a20 0x7f6870106380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:20:50.711 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.174+0000 7f6875945700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68701068c0 con 0x7f6870106a20 2023-07-20T01:20:50.712 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.174+0000 7f686effd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6870106a20 0x7f6870106380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:20:50.712 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.174+0000 7f686effd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6870106a20 0x7f6870106380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41804/0 (socket says 172.21.15.60:41804) 2023-07-20T01:20:50.712 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.174+0000 7f686effd700 1 -- 172.21.15.60:0/1442923720 learned_addr learned my addr 172.21.15.60:0/1442923720 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:20:50.712 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.175+0000 7f686effd700 1 -- 172.21.15.60:0/1442923720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6860009550 con 0x7f6870106a20 2023-07-20T01:20:50.712 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.175+0000 7f686effd700 1 --2- 172.21.15.60:0/1442923720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6870106a20 0x7f6870106380 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6860011dc0 tx=0x7f6860011ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:20:50.712 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.175+0000 7f6874943700 1 -- 172.21.15.60:0/1442923720 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f68600037d0 con 0x7f6870106a20 2023-07-20T01:20:50.712 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.175+0000 7f6874943700 1 -- 172.21.15.60:0/1442923720 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f686001b440 con 0x7f6870106a20 2023-07-20T01:20:50.713 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.175+0000 7f6874943700 1 -- 172.21.15.60:0/1442923720 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f686001a560 con 0x7f6870106a20 2023-07-20T01:20:50.713 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.175+0000 7f6875945700 1 -- 172.21.15.60:0/1442923720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68701049d0 con 0x7f6870106a20 2023-07-20T01:20:50.713 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.175+0000 7f6875945700 1 -- 172.21.15.60:0/1442923720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6870104df0 con 0x7f6870106a20 2023-07-20T01:20:50.713 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.175+0000 7f6874943700 1 -- 172.21.15.60:0/1442923720 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f686001aad0 con 0x7f6870106a20 2023-07-20T01:20:50.713 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.175+0000 7f6874943700 1 -- 172.21.15.60:0/1442923720 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f68600193f0 con 0x7f6870106a20 2023-07-20T01:20:50.713 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.176+0000 7f6875945700 1 -- 172.21.15.60:0/1442923720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f685c005320 con 0x7f6870106a20 2023-07-20T01:20:50.713 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.178+0000 7f6874943700 1 -- 172.21.15.60:0/1442923720 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f6860029060 con 0x7f6870106a20 2023-07-20T01:20:50.714 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.311+0000 7f6875945700 1 -- 172.21.15.60:0/1442923720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f685c005cc0 con 0x7f6870106a20 2023-07-20T01:20:50.714 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.312+0000 7f6874943700 1 -- 172.21.15.60:0/1442923720 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f686001f070 con 0x7f6870106a20 2023-07-20T01:20:50.714 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.313+0000 7f6875945700 1 -- 172.21.15.60:0/1442923720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6870106a20 msgr2=0x7f6870106380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:20:50.714 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.313+0000 7f6875945700 1 --2- 172.21.15.60:0/1442923720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6870106a20 0x7f6870106380 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6860011dc0 tx=0x7f6860011ea0 comp rx=0 tx=0).stop 2023-07-20T01:20:50.714 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.313+0000 7f6875945700 1 -- 172.21.15.60:0/1442923720 shutdown_connections 2023-07-20T01:20:50.714 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.313+0000 7f6875945700 1 --2- 172.21.15.60:0/1442923720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6870106a20 0x7f6870106380 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:20:50.714 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.313+0000 7f6875945700 1 -- 172.21.15.60:0/1442923720 >> 172.21.15.60:0/1442923720 conn(0x7f6870100490 msgr2=0x7f6870101250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:20:50.715 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.313+0000 7f6875945700 1 -- 172.21.15.60:0/1442923720 shutdown_connections 2023-07-20T01:20:50.715 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:50.313+0000 7f6875945700 1 -- 172.21.15.60:0/1442923720 wait complete. 2023-07-20T01:20:50.715 INFO:teuthology.orchestra.run.smithi060.stdout:mgr not available, waiting (3/15)... 2023-07-20T01:20:51.680 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:20:51 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:20:51.291+0000 7f3c9be43200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T01:20:51.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:51 smithi060 ceph-mon[96524]: Activating manager daemon x 2023-07-20T01:20:51.681 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:51 smithi060 ceph-mon[96524]: mgrmap e2: x(active, starting, since 0.0031847s) 2023-07-20T01:20:51.681 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:51 smithi060 ceph-mon[96524]: from='mgr.14100 172.21.15.60:0/2828868019' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-20T01:20:51.681 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:51 smithi060 ceph-mon[96524]: from='mgr.14100 172.21.15.60:0/2828868019' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-20T01:20:51.682 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:51 smithi060 ceph-mon[96524]: from='mgr.14100 172.21.15.60:0/2828868019' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-20T01:20:51.682 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:51 smithi060 ceph-mon[96524]: from='mgr.14100 172.21.15.60:0/2828868019' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:20:51.682 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:51 smithi060 ceph-mon[96524]: from='mgr.14100 172.21.15.60:0/2828868019' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-20T01:20:51.682 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:51 smithi060 ceph-mon[96524]: Manager daemon x is now available 2023-07-20T01:20:51.683 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:51 smithi060 ceph-mon[96524]: from='mgr.14100 172.21.15.60:0/2828868019' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-20T01:20:51.683 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:51 smithi060 ceph-mon[96524]: from='mgr.14100 172.21.15.60:0/2828868019' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-20T01:20:51.683 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:51 smithi060 ceph-mon[96524]: from='mgr.14100 172.21.15.60:0/2828868019' entity='mgr.x' 2023-07-20T01:20:51.684 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:51 smithi060 ceph-mon[96524]: from='mgr.14100 172.21.15.60:0/2828868019' entity='mgr.x' 2023-07-20T01:20:51.684 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:51 smithi060 ceph-mon[96524]: from='mgr.14100 172.21.15.60:0/2828868019' entity='mgr.x' 2023-07-20T01:20:53.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:53 smithi060 ceph-mon[96524]: mgrmap e3: x(active, since 1.00993s) 2023-07-20T01:20:55.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:55 smithi060 ceph-mon[96524]: mgrmap e4: x(active, since 3s) 2023-07-20T01:20:55.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:20:55 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/2988044785' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-20T01:20:58.163 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 2023-07-20T01:20:58.164 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout { 2023-07-20T01:20:58.164 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "fsid": "7daf031e-269b-11ee-9b34-001a4aab830c", 2023-07-20T01:20:58.164 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "health": { 2023-07-20T01:20:58.164 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-20T01:20:58.165 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-20T01:20:58.165 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-20T01:20:58.165 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:20:58.165 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-20T01:20:58.165 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-20T01:20:58.165 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 0 2023-07-20T01:20:58.166 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout ], 2023-07-20T01:20:58.166 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-20T01:20:58.166 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "a" 2023-07-20T01:20:58.166 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout ], 2023-07-20T01:20:58.166 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2023-07-20T01:20:58.166 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-20T01:20:58.166 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T01:20:58.166 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-20T01:20:58.167 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-20T01:20:58.167 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:20:58.167 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-20T01:20:58.167 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T01:20:58.167 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-20T01:20:58.167 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-20T01:20:58.167 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-20T01:20:58.167 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-20T01:20:58.168 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-20T01:20:58.168 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-20T01:20:58.168 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:20:58.168 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-20T01:20:58.168 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-20T01:20:58.168 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-20T01:20:58.168 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-20T01:20:58.169 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-20T01:20:58.169 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-20T01:20:58.169 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-20T01:20:58.169 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-20T01:20:58.169 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-20T01:20:58.169 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:20:58.170 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-20T01:20:58.170 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T01:20:58.170 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-20T01:20:58.170 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-20T01:20:58.170 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:20:58.170 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-20T01:20:58.170 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-20T01:20:58.171 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-20T01:20:58.171 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-20T01:20:58.171 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-20T01:20:58.171 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-20T01:20:58.171 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "restful" 2023-07-20T01:20:58.172 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout ], 2023-07-20T01:20:58.172 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T01:20:58.172 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:20:58.172 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-20T01:20:58.173 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-20T01:20:58.173 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "modified": "2023-07-20T01:20:36.943821+0000", 2023-07-20T01:20:58.173 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-20T01:20:58.173 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2023-07-20T01:20:58.174 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-20T01:20:58.174 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout } 2023-07-20T01:20:58.174 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.596+0000 7f38797c6700 1 Processor -- start 2023-07-20T01:20:58.174 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.596+0000 7f38797c6700 1 -- start start 2023-07-20T01:20:58.174 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.597+0000 7f38797c6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3874106a20 0x7f3874108e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:20:58.174 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.597+0000 7f38797c6700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3874079e50 con 0x7f3874106a20 2023-07-20T01:20:58.174 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.597+0000 7f3872ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3874106a20 0x7f3874108e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:20:58.175 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.597+0000 7f3872ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3874106a20 0x7f3874108e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41830/0 (socket says 172.21.15.60:41830) 2023-07-20T01:20:58.175 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.597+0000 7f3872ffd700 1 -- 172.21.15.60:0/3974767811 learned_addr learned my addr 172.21.15.60:0/3974767811 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:20:58.175 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.597+0000 7f3872ffd700 1 -- 172.21.15.60:0/3974767811 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3874109350 con 0x7f3874106a20 2023-07-20T01:20:58.175 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.597+0000 7f3872ffd700 1 --2- 172.21.15.60:0/3974767811 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3874106a20 0x7f3874108e10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f38640098a0 tx=0x7f3864009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71eccda41bb01a82 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:20:58.175 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.598+0000 7f3871ffb700 1 -- 172.21.15.60:0/3974767811 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3864004160 con 0x7f3874106a20 2023-07-20T01:20:58.175 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.598+0000 7f3871ffb700 1 -- 172.21.15.60:0/3974767811 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f38640042c0 con 0x7f3874106a20 2023-07-20T01:20:58.176 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.598+0000 7f3871ffb700 1 -- 172.21.15.60:0/3974767811 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3864011940 con 0x7f3874106a20 2023-07-20T01:20:58.176 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.598+0000 7f38797c6700 1 -- 172.21.15.60:0/3974767811 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3874106a20 msgr2=0x7f3874108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:20:58.176 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.598+0000 7f38797c6700 1 --2- 172.21.15.60:0/3974767811 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3874106a20 0x7f3874108e10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f38640098a0 tx=0x7f3864009bb0 comp rx=0 tx=0).stop 2023-07-20T01:20:58.176 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.598+0000 7f38797c6700 1 -- 172.21.15.60:0/3974767811 shutdown_connections 2023-07-20T01:20:58.176 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.598+0000 7f38797c6700 1 --2- 172.21.15.60:0/3974767811 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3874106a20 0x7f3874108e10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:20:58.176 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.598+0000 7f38797c6700 1 -- 172.21.15.60:0/3974767811 >> 172.21.15.60:0/3974767811 conn(0x7f3874100470 msgr2=0x7f3874102890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:20:58.176 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.598+0000 7f38797c6700 1 -- 172.21.15.60:0/3974767811 shutdown_connections 2023-07-20T01:20:58.177 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.598+0000 7f38797c6700 1 -- 172.21.15.60:0/3974767811 wait complete. 2023-07-20T01:20:58.177 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.599+0000 7f38797c6700 1 Processor -- start 2023-07-20T01:20:58.177 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.599+0000 7f38797c6700 1 -- start start 2023-07-20T01:20:58.177 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.600+0000 7f38797c6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3874106a20 0x7f3874199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:20:58.177 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.600+0000 7f38797c6700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f387419a0f0 con 0x7f3874106a20 2023-07-20T01:20:58.177 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.600+0000 7f3872ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3874106a20 0x7f3874199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:20:58.177 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.600+0000 7f3872ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3874106a20 0x7f3874199bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41832/0 (socket says 172.21.15.60:41832) 2023-07-20T01:20:58.177 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.600+0000 7f3872ffd700 1 -- 172.21.15.60:0/2988044785 learned_addr learned my addr 172.21.15.60:0/2988044785 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:20:58.178 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.600+0000 7f3872ffd700 1 -- 172.21.15.60:0/2988044785 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3864009550 con 0x7f3874106a20 2023-07-20T01:20:58.178 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.600+0000 7f3872ffd700 1 --2- 172.21.15.60:0/2988044785 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3874106a20 0x7f3874199bb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3864006af0 tx=0x7f3864011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:20:58.178 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.600+0000 7f385bfff700 1 -- 172.21.15.60:0/2988044785 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3864003bf0 con 0x7f3874106a20 2023-07-20T01:20:58.178 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.601+0000 7f385bfff700 1 -- 172.21.15.60:0/2988044785 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f386401b8b0 con 0x7f3874106a20 2023-07-20T01:20:58.178 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.601+0000 7f38797c6700 1 -- 172.21.15.60:0/2988044785 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f387419a2f0 con 0x7f3874106a20 2023-07-20T01:20:58.178 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.601+0000 7f385bfff700 1 -- 172.21.15.60:0/2988044785 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3864019880 con 0x7f3874106a20 2023-07-20T01:20:58.178 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.601+0000 7f38797c6700 1 -- 172.21.15.60:0/2988044785 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f387419a710 con 0x7f3874106a20 2023-07-20T01:20:58.179 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.602+0000 7f38797c6700 1 -- 172.21.15.60:0/2988044785 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f387410a0f0 con 0x7f3874106a20 2023-07-20T01:20:58.179 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.602+0000 7f385bfff700 1 -- 172.21.15.60:0/2988044785 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f386401b440 con 0x7f3874106a20 2023-07-20T01:20:58.179 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.602+0000 7f385bfff700 1 --2- 172.21.15.60:0/2988044785 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7f385c038670 0x7f385c03ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:20:58.179 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.602+0000 7f385bfff700 1 -- 172.21.15.60:0/2988044785 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f386404c9a0 con 0x7f3874106a20 2023-07-20T01:20:58.179 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.602+0000 7f38727fc700 1 --2- 172.21.15.60:0/2988044785 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7f385c038670 0x7f385c03ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:20:58.179 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.603+0000 7f38727fc700 1 --2- 172.21.15.60:0/2988044785 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7f385c038670 0x7f385c03ab30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3868006fd0 tx=0x7f3868006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:20:58.180 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:54.608+0000 7f385bfff700 1 -- 172.21.15.60:0/2988044785 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f386401b6f0 con 0x7f3874106a20 2023-07-20T01:20:58.180 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:55.121+0000 7f38797c6700 1 -- 172.21.15.60:0/2988044785 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f387410a2e0 con 0x7f3874106a20 2023-07-20T01:20:58.180 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:55.122+0000 7f385bfff700 1 -- 172.21.15.60:0/2988044785 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f38640281c0 con 0x7f3874106a20 2023-07-20T01:20:58.180 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:55.124+0000 7f38797c6700 1 -- 172.21.15.60:0/2988044785 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7f385c038670 msgr2=0x7f385c03ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:20:58.180 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:55.124+0000 7f38797c6700 1 --2- 172.21.15.60:0/2988044785 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7f385c038670 0x7f385c03ab30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3868006fd0 tx=0x7f3868006e40 comp rx=0 tx=0).stop 2023-07-20T01:20:58.180 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:55.124+0000 7f38797c6700 1 -- 172.21.15.60:0/2988044785 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3874106a20 msgr2=0x7f3874199bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:20:58.180 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:55.124+0000 7f38797c6700 1 --2- 172.21.15.60:0/2988044785 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3874106a20 0x7f3874199bb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3864006af0 tx=0x7f3864011a30 comp rx=0 tx=0).stop 2023-07-20T01:20:58.181 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:55.124+0000 7f38797c6700 1 -- 172.21.15.60:0/2988044785 shutdown_connections 2023-07-20T01:20:58.181 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:55.124+0000 7f38797c6700 1 --2- 172.21.15.60:0/2988044785 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7f385c038670 0x7f385c03ab30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:20:58.181 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:55.124+0000 7f38797c6700 1 --2- 172.21.15.60:0/2988044785 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3874106a20 0x7f3874199bb0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:20:58.181 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:55.124+0000 7f38797c6700 1 -- 172.21.15.60:0/2988044785 >> 172.21.15.60:0/2988044785 conn(0x7f3874100470 msgr2=0x7f3874101200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:20:58.181 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:55.124+0000 7f38797c6700 1 -- 172.21.15.60:0/2988044785 shutdown_connections 2023-07-20T01:20:58.181 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:20:55.124+0000 7f38797c6700 1 -- 172.21.15.60:0/2988044785 wait complete. 2023-07-20T01:20:58.181 INFO:teuthology.orchestra.run.smithi060.stdout:mgr is available 2023-07-20T01:21:00.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:00 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/3493697314' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-20T01:21:02.319 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 2023-07-20T01:21:02.324 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout [global] 2023-07-20T01:21:02.325 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout fsid = 7daf031e-269b-11ee-9b34-001a4aab830c 2023-07-20T01:21:02.325 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-20T01:21:02.325 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-20T01:21:02.325 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-20T01:21:02.325 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-20T01:21:02.326 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 2023-07-20T01:21:02.326 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-20T01:21:02.326 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-20T01:21:02.326 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 2023-07-20T01:21:02.326 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout [osd] 2023-07-20T01:21:02.326 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2023-07-20T01:21:02.326 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-07-20T01:21:02.326 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-07-20T01:21:02.327 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-07-20T01:21:02.327 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-20T01:21:02.327 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-20T01:21:02.327 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.159+0000 7fec3936f700 1 Processor -- start 2023-07-20T01:21:02.327 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.159+0000 7fec3936f700 1 -- start start 2023-07-20T01:21:02.327 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.159+0000 7fec3936f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec34106b00 0x7fec34108ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:02.327 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.159+0000 7fec3936f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec34073e20 con 0x7fec34106b00 2023-07-20T01:21:02.327 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.160+0000 7fec32ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec34106b00 0x7fec34108ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:02.328 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.160+0000 7fec32ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec34106b00 0x7fec34108ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41836/0 (socket says 172.21.15.60:41836) 2023-07-20T01:21:02.328 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.160+0000 7fec32ffd700 1 -- 172.21.15.60:0/2098986049 learned_addr learned my addr 172.21.15.60:0/2098986049 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:02.328 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.160+0000 7fec32ffd700 1 -- 172.21.15.60:0/2098986049 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec34073f60 con 0x7fec34106b00 2023-07-20T01:21:02.328 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.160+0000 7fec32ffd700 1 --2- 172.21.15.60:0/2098986049 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec34106b00 0x7fec34108ef0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fec280098a0 tx=0x7fec28009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d042fb44f9f4570f server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:02.328 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.160+0000 7fec31ffb700 1 -- 172.21.15.60:0/2098986049 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec28004160 con 0x7fec34106b00 2023-07-20T01:21:02.328 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.160+0000 7fec31ffb700 1 -- 172.21.15.60:0/2098986049 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fec280042c0 con 0x7fec34106b00 2023-07-20T01:21:02.328 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.160+0000 7fec31ffb700 1 -- 172.21.15.60:0/2098986049 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec28011940 con 0x7fec34106b00 2023-07-20T01:21:02.329 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.161+0000 7fec3936f700 1 -- 172.21.15.60:0/2098986049 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec34106b00 msgr2=0x7fec34108ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:02.329 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.161+0000 7fec3936f700 1 --2- 172.21.15.60:0/2098986049 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec34106b00 0x7fec34108ef0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fec280098a0 tx=0x7fec28009bb0 comp rx=0 tx=0).stop 2023-07-20T01:21:02.329 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.161+0000 7fec3936f700 1 -- 172.21.15.60:0/2098986049 shutdown_connections 2023-07-20T01:21:02.329 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.161+0000 7fec3936f700 1 --2- 172.21.15.60:0/2098986049 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec34106b00 0x7fec34108ef0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:02.329 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.161+0000 7fec3936f700 1 -- 172.21.15.60:0/2098986049 >> 172.21.15.60:0/2098986049 conn(0x7fec34100490 msgr2=0x7fec341028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:02.329 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.161+0000 7fec3936f700 1 -- 172.21.15.60:0/2098986049 shutdown_connections 2023-07-20T01:21:02.329 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.161+0000 7fec3936f700 1 -- 172.21.15.60:0/2098986049 wait complete. 2023-07-20T01:21:02.329 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.162+0000 7fec3936f700 1 Processor -- start 2023-07-20T01:21:02.330 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.162+0000 7fec3936f700 1 -- start start 2023-07-20T01:21:02.330 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.162+0000 7fec3936f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec34106b00 0x7fec34199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:02.330 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.162+0000 7fec3936f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3419a110 con 0x7fec34106b00 2023-07-20T01:21:02.330 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.162+0000 7fec32ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec34106b00 0x7fec34199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:02.330 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.162+0000 7fec32ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec34106b00 0x7fec34199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41838/0 (socket says 172.21.15.60:41838) 2023-07-20T01:21:02.330 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.162+0000 7fec32ffd700 1 -- 172.21.15.60:0/3493697314 learned_addr learned my addr 172.21.15.60:0/3493697314 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:02.330 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.162+0000 7fec32ffd700 1 -- 172.21.15.60:0/3493697314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec28009550 con 0x7fec34106b00 2023-07-20T01:21:02.330 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.163+0000 7fec32ffd700 1 --2- 172.21.15.60:0/3493697314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec34106b00 0x7fec34199bd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fec28006af0 tx=0x7fec28011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:02.331 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.163+0000 7fec1bfff700 1 -- 172.21.15.60:0/3493697314 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec28003bf0 con 0x7fec34106b00 2023-07-20T01:21:02.331 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.163+0000 7fec1bfff700 1 -- 172.21.15.60:0/3493697314 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fec2801b8b0 con 0x7fec34106b00 2023-07-20T01:21:02.331 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.163+0000 7fec1bfff700 1 -- 172.21.15.60:0/3493697314 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec28019880 con 0x7fec34106b00 2023-07-20T01:21:02.331 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.163+0000 7fec3936f700 1 -- 172.21.15.60:0/3493697314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec3419a310 con 0x7fec34106b00 2023-07-20T01:21:02.331 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.163+0000 7fec3936f700 1 -- 172.21.15.60:0/3493697314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec3419a730 con 0x7fec34106b00 2023-07-20T01:21:02.331 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.164+0000 7fec1bfff700 1 -- 172.21.15.60:0/3493697314 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7fec2801b440 con 0x7fec34106b00 2023-07-20T01:21:02.331 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.164+0000 7fec1bfff700 1 --2- 172.21.15.60:0/3493697314 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7fec1c0386c0 0x7fec1c03ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:02.332 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.164+0000 7fec1bfff700 1 -- 172.21.15.60:0/3493697314 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fec2804c910 con 0x7fec34106b00 2023-07-20T01:21:02.332 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.164+0000 7fec3936f700 1 -- 172.21.15.60:0/3493697314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec3410a100 con 0x7fec34106b00 2023-07-20T01:21:02.332 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.164+0000 7fec327fc700 1 --2- 172.21.15.60:0/3493697314 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7fec1c0386c0 0x7fec1c03ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:02.332 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.164+0000 7fec327fc700 1 --2- 172.21.15.60:0/3493697314 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7fec1c0386c0 0x7fec1c03ab80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fec24006fd0 tx=0x7fec24006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:02.332 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.169+0000 7fec1bfff700 1 -- 172.21.15.60:0/3493697314 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fec28019af0 con 0x7fec34106b00 2023-07-20T01:21:02.332 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.527+0000 7fec3936f700 1 -- 172.21.15.60:0/3493697314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fec3410a2f0 con 0x7fec34106b00 2023-07-20T01:21:02.332 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.528+0000 7fec1bfff700 1 -- 172.21.15.60:0/3493697314 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+398 (secure 0 0 0) 0x7fec2801f250 con 0x7fec34106b00 2023-07-20T01:21:02.332 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.530+0000 7fec3936f700 1 -- 172.21.15.60:0/3493697314 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7fec1c0386c0 msgr2=0x7fec1c03ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:02.333 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.530+0000 7fec3936f700 1 --2- 172.21.15.60:0/3493697314 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7fec1c0386c0 0x7fec1c03ab80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fec24006fd0 tx=0x7fec24006e40 comp rx=0 tx=0).stop 2023-07-20T01:21:02.333 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.530+0000 7fec3936f700 1 -- 172.21.15.60:0/3493697314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec34106b00 msgr2=0x7fec34199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:02.333 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.530+0000 7fec3936f700 1 --2- 172.21.15.60:0/3493697314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec34106b00 0x7fec34199bd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fec28006af0 tx=0x7fec28011a30 comp rx=0 tx=0).stop 2023-07-20T01:21:02.333 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.530+0000 7fec3936f700 1 -- 172.21.15.60:0/3493697314 shutdown_connections 2023-07-20T01:21:02.333 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.530+0000 7fec3936f700 1 --2- 172.21.15.60:0/3493697314 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7fec1c0386c0 0x7fec1c03ab80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:02.333 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.530+0000 7fec3936f700 1 --2- 172.21.15.60:0/3493697314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec34106b00 0x7fec34199bd0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:02.333 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.530+0000 7fec3936f700 1 -- 172.21.15.60:0/3493697314 >> 172.21.15.60:0/3493697314 conn(0x7fec34100490 msgr2=0x7fec34101270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:02.334 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.530+0000 7fec3936f700 1 -- 172.21.15.60:0/3493697314 shutdown_connections 2023-07-20T01:21:02.334 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:00.530+0000 7fec3936f700 1 -- 172.21.15.60:0/3493697314 wait complete. 2023-07-20T01:21:02.334 INFO:teuthology.orchestra.run.smithi060.stdout:Enabling cephadm module... 2023-07-20T01:21:03.647 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:03 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/2139373269' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-07-20T01:21:03.648 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:03 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: ignoring --setuser ceph since I am not root 2023-07-20T01:21:03.648 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:03 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: ignoring --setgroup ceph since I am not root 2023-07-20T01:21:03.648 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:03 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:03.567+0000 7f73f279c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T01:21:03.649 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:03 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:03.664+0000 7f73f279c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T01:21:03.775 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.853+0000 7f90b81d4700 1 Processor -- start 2023-07-20T01:21:03.776 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.853+0000 7f90b81d4700 1 -- start start 2023-07-20T01:21:03.776 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.853+0000 7f90b81d4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90b0106a30 0x7f90b0108e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:03.776 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.853+0000 7f90b81d4700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b0073f00 con 0x7f90b0106a30 2023-07-20T01:21:03.776 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.853+0000 7f90b5f70700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90b0106a30 0x7f90b0108e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:03.776 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.853+0000 7f90b5f70700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90b0106a30 0x7f90b0108e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41842/0 (socket says 172.21.15.60:41842) 2023-07-20T01:21:03.776 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.853+0000 7f90b5f70700 1 -- 172.21.15.60:0/2349140412 learned_addr learned my addr 172.21.15.60:0/2349140412 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:03.776 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.854+0000 7f90b5f70700 1 -- 172.21.15.60:0/2349140412 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b0109360 con 0x7f90b0106a30 2023-07-20T01:21:03.777 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.854+0000 7f90b5f70700 1 --2- 172.21.15.60:0/2349140412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90b0106a30 0x7f90b0108e20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f90a40098a0 tx=0x7f90a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=770e71e04e748ec3 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:03.777 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.854+0000 7f90b4f6e700 1 -- 172.21.15.60:0/2349140412 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f90a4004160 con 0x7f90b0106a30 2023-07-20T01:21:03.777 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.854+0000 7f90b4f6e700 1 -- 172.21.15.60:0/2349140412 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f90a40042c0 con 0x7f90b0106a30 2023-07-20T01:21:03.777 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.854+0000 7f90b4f6e700 1 -- 172.21.15.60:0/2349140412 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f90a4011940 con 0x7f90b0106a30 2023-07-20T01:21:03.777 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.854+0000 7f90b81d4700 1 -- 172.21.15.60:0/2349140412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90b0106a30 msgr2=0x7f90b0108e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:03.777 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.854+0000 7f90b81d4700 1 --2- 172.21.15.60:0/2349140412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90b0106a30 0x7f90b0108e20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f90a40098a0 tx=0x7f90a4009bb0 comp rx=0 tx=0).stop 2023-07-20T01:21:03.777 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.854+0000 7f90b81d4700 1 -- 172.21.15.60:0/2349140412 shutdown_connections 2023-07-20T01:21:03.778 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.854+0000 7f90b81d4700 1 --2- 172.21.15.60:0/2349140412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90b0106a30 0x7f90b0108e20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:03.778 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.855+0000 7f90b81d4700 1 -- 172.21.15.60:0/2349140412 >> 172.21.15.60:0/2349140412 conn(0x7f90b01004a0 msgr2=0x7f90b01028c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:03.778 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.855+0000 7f90b81d4700 1 -- 172.21.15.60:0/2349140412 shutdown_connections 2023-07-20T01:21:03.778 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.855+0000 7f90b81d4700 1 -- 172.21.15.60:0/2349140412 wait complete. 2023-07-20T01:21:03.778 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.855+0000 7f90b81d4700 1 Processor -- start 2023-07-20T01:21:03.778 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.855+0000 7f90b81d4700 1 -- start start 2023-07-20T01:21:03.778 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.855+0000 7f90b81d4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90b0106a30 0x7f90b0199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:03.778 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.855+0000 7f90b81d4700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b019a100 con 0x7f90b0106a30 2023-07-20T01:21:03.779 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.856+0000 7f90b5f70700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90b0106a30 0x7f90b0199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:03.779 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.856+0000 7f90b5f70700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90b0106a30 0x7f90b0199bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41844/0 (socket says 172.21.15.60:41844) 2023-07-20T01:21:03.779 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.856+0000 7f90b5f70700 1 -- 172.21.15.60:0/2139373269 learned_addr learned my addr 172.21.15.60:0/2139373269 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:03.779 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.856+0000 7f90b5f70700 1 -- 172.21.15.60:0/2139373269 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a4009550 con 0x7f90b0106a30 2023-07-20T01:21:03.779 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.856+0000 7f90b5f70700 1 --2- 172.21.15.60:0/2139373269 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90b0106a30 0x7f90b0199bc0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f90a4006af0 tx=0x7f90a4011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:03.779 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.856+0000 7f90a2ffd700 1 -- 172.21.15.60:0/2139373269 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f90a4003bf0 con 0x7f90b0106a30 2023-07-20T01:21:03.779 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.856+0000 7f90a2ffd700 1 -- 172.21.15.60:0/2139373269 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f90a401b8b0 con 0x7f90b0106a30 2023-07-20T01:21:03.780 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.856+0000 7f90b81d4700 1 -- 172.21.15.60:0/2139373269 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90b019a300 con 0x7f90b0106a30 2023-07-20T01:21:03.780 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.856+0000 7f90a2ffd700 1 -- 172.21.15.60:0/2139373269 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f90a40197b0 con 0x7f90b0106a30 2023-07-20T01:21:03.780 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.856+0000 7f90b81d4700 1 -- 172.21.15.60:0/2139373269 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90b019a720 con 0x7f90b0106a30 2023-07-20T01:21:03.780 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.857+0000 7f90a2ffd700 1 -- 172.21.15.60:0/2139373269 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f90a401b440 con 0x7f90b0106a30 2023-07-20T01:21:03.780 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.857+0000 7f90b81d4700 1 -- 172.21.15.60:0/2139373269 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90b010a100 con 0x7f90b0106a30 2023-07-20T01:21:03.780 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.857+0000 7f90a2ffd700 1 --2- 172.21.15.60:0/2139373269 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7f909c0386c0 0x7f909c03ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:03.780 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.857+0000 7f90a2ffd700 1 -- 172.21.15.60:0/2139373269 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f90a404c940 con 0x7f90b0106a30 2023-07-20T01:21:03.780 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.858+0000 7f90b576f700 1 --2- 172.21.15.60:0/2139373269 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7f909c0386c0 0x7f909c03ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:03.781 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.858+0000 7f90b576f700 1 --2- 172.21.15.60:0/2139373269 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7f909c0386c0 0x7f909c03ab80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f90ac006fd0 tx=0x7f90ac006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:03.781 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:02.862+0000 7f90a2ffd700 1 -- 172.21.15.60:0/2139373269 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f90a401bd70 con 0x7f90b0106a30 2023-07-20T01:21:03.781 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:03.328+0000 7f90b81d4700 1 -- 172.21.15.60:0/2139373269 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f90b010a320 con 0x7f90b0106a30 2023-07-20T01:21:03.781 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:03.381+0000 7f90a2ffd700 1 -- 172.21.15.60:0/2139373269 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f90a400f2e0 con 0x7f90b0106a30 2023-07-20T01:21:03.781 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:03.382+0000 7f90a2ffd700 1 -- 172.21.15.60:0/2139373269 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f90a402bb00 con 0x7f90b0106a30 2023-07-20T01:21:03.781 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:03.384+0000 7f90b81d4700 1 -- 172.21.15.60:0/2139373269 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7f909c0386c0 msgr2=0x7f909c03ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:03.781 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:03.384+0000 7f90b81d4700 1 --2- 172.21.15.60:0/2139373269 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7f909c0386c0 0x7f909c03ab80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f90ac006fd0 tx=0x7f90ac006e40 comp rx=0 tx=0).stop 2023-07-20T01:21:03.782 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:03.384+0000 7f90b81d4700 1 -- 172.21.15.60:0/2139373269 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90b0106a30 msgr2=0x7f90b0199bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:03.782 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:03.384+0000 7f90b81d4700 1 --2- 172.21.15.60:0/2139373269 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90b0106a30 0x7f90b0199bc0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f90a4006af0 tx=0x7f90a4011a30 comp rx=0 tx=0).stop 2023-07-20T01:21:03.782 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:03.384+0000 7f90b81d4700 1 -- 172.21.15.60:0/2139373269 shutdown_connections 2023-07-20T01:21:03.782 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:03.384+0000 7f90b81d4700 1 --2- 172.21.15.60:0/2139373269 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7f909c0386c0 0x7f909c03ab80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:03.782 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:03.384+0000 7f90b81d4700 1 --2- 172.21.15.60:0/2139373269 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90b0106a30 0x7f90b0199bc0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:03.782 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:03.384+0000 7f90b81d4700 1 -- 172.21.15.60:0/2139373269 >> 172.21.15.60:0/2139373269 conn(0x7f90b01004a0 msgr2=0x7f90b0101270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:03.782 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:03.384+0000 7f90b81d4700 1 -- 172.21.15.60:0/2139373269 shutdown_connections 2023-07-20T01:21:03.782 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:03.385+0000 7f90b81d4700 1 -- 172.21.15.60:0/2139373269 wait complete. 2023-07-20T01:21:03.929 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:03 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:03.773+0000 7f73f279c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T01:21:03.930 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:03 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:03.892+0000 7f73f279c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T01:21:04.366 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:04 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:04.106+0000 7f73f279c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T01:21:04.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:04 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/2139373269' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-07-20T01:21:04.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:04 smithi060 ceph-mon[96524]: mgrmap e5: x(active, since 12s) 2023-07-20T01:21:04.680 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:04 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:04.508+0000 7f73f279c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T01:21:04.680 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:04 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:04.617+0000 7f73f279c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T01:21:05.015 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:04 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:04.769+0000 7f73f279c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T01:21:05.016 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:04 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:04.890+0000 7f73f279c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T01:21:05.020 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout { 2023-07-20T01:21:05.021 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-07-20T01:21:05.021 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-20T01:21:05.021 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-07-20T01:21:05.021 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-20T01:21:05.021 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout } 2023-07-20T01:21:05.021 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.262+0000 7f91a9716700 1 Processor -- start 2023-07-20T01:21:05.022 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.262+0000 7f91a9716700 1 -- start start 2023-07-20T01:21:05.022 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.262+0000 7f91a9716700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91a4106a20 0x7f91a4108e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:05.022 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.262+0000 7f91a9716700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a4073f00 con 0x7f91a4106a20 2023-07-20T01:21:05.022 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.262+0000 7f91a2ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91a4106a20 0x7f91a4108e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:05.022 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.262+0000 7f91a2ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91a4106a20 0x7f91a4108e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41858/0 (socket says 172.21.15.60:41858) 2023-07-20T01:21:05.022 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.262+0000 7f91a2ffd700 1 -- 172.21.15.60:0/918918172 learned_addr learned my addr 172.21.15.60:0/918918172 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:05.022 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.263+0000 7f91a2ffd700 1 -- 172.21.15.60:0/918918172 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a4109350 con 0x7f91a4106a20 2023-07-20T01:21:05.023 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.263+0000 7f91a2ffd700 1 --2- 172.21.15.60:0/918918172 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91a4106a20 0x7f91a4108e10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f91940098a0 tx=0x7f9194009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4a8c56c15d21bb8 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:05.023 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.263+0000 7f91a1ffb700 1 -- 172.21.15.60:0/918918172 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9194004160 con 0x7f91a4106a20 2023-07-20T01:21:05.023 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.263+0000 7f91a1ffb700 1 -- 172.21.15.60:0/918918172 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f91940042c0 con 0x7f91a4106a20 2023-07-20T01:21:05.023 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.263+0000 7f91a1ffb700 1 -- 172.21.15.60:0/918918172 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9194011940 con 0x7f91a4106a20 2023-07-20T01:21:05.023 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.264+0000 7f91a9716700 1 -- 172.21.15.60:0/918918172 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91a4106a20 msgr2=0x7f91a4108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:05.023 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.264+0000 7f91a9716700 1 --2- 172.21.15.60:0/918918172 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91a4106a20 0x7f91a4108e10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f91940098a0 tx=0x7f9194009bb0 comp rx=0 tx=0).stop 2023-07-20T01:21:05.023 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.264+0000 7f91a9716700 1 -- 172.21.15.60:0/918918172 shutdown_connections 2023-07-20T01:21:05.023 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.264+0000 7f91a9716700 1 --2- 172.21.15.60:0/918918172 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91a4106a20 0x7f91a4108e10 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:05.024 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.264+0000 7f91a9716700 1 -- 172.21.15.60:0/918918172 >> 172.21.15.60:0/918918172 conn(0x7f91a4100490 msgr2=0x7f91a41028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:05.024 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.264+0000 7f91a9716700 1 -- 172.21.15.60:0/918918172 shutdown_connections 2023-07-20T01:21:05.024 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.264+0000 7f91a9716700 1 -- 172.21.15.60:0/918918172 wait complete. 2023-07-20T01:21:05.024 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.264+0000 7f91a9716700 1 Processor -- start 2023-07-20T01:21:05.024 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.265+0000 7f91a9716700 1 -- start start 2023-07-20T01:21:05.024 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.265+0000 7f91a9716700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91a4106a20 0x7f91a4199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:05.025 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.265+0000 7f91a9716700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a419a0e0 con 0x7f91a4106a20 2023-07-20T01:21:05.025 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.265+0000 7f91a2ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91a4106a20 0x7f91a4199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:05.025 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.265+0000 7f91a2ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91a4106a20 0x7f91a4199ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41860/0 (socket says 172.21.15.60:41860) 2023-07-20T01:21:05.025 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.265+0000 7f91a2ffd700 1 -- 172.21.15.60:0/2342584965 learned_addr learned my addr 172.21.15.60:0/2342584965 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:05.025 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.265+0000 7f91a2ffd700 1 -- 172.21.15.60:0/2342584965 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9194009550 con 0x7f91a4106a20 2023-07-20T01:21:05.025 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.265+0000 7f91a2ffd700 1 --2- 172.21.15.60:0/2342584965 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91a4106a20 0x7f91a4199ba0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9194011dc0 tx=0x7f9194011ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:05.025 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.265+0000 7f918bfff700 1 -- 172.21.15.60:0/2342584965 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f91940037d0 con 0x7f91a4106a20 2023-07-20T01:21:05.026 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.266+0000 7f918bfff700 1 -- 172.21.15.60:0/2342584965 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f919401b440 con 0x7f91a4106a20 2023-07-20T01:21:05.026 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.266+0000 7f91a9716700 1 -- 172.21.15.60:0/2342584965 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91a419a2e0 con 0x7f91a4106a20 2023-07-20T01:21:05.026 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.266+0000 7f918bfff700 1 -- 172.21.15.60:0/2342584965 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f919401a560 con 0x7f91a4106a20 2023-07-20T01:21:05.026 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.266+0000 7f91a9716700 1 -- 172.21.15.60:0/2342584965 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91a419a700 con 0x7f91a4106a20 2023-07-20T01:21:05.026 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.267+0000 7f918bfff700 1 -- 172.21.15.60:0/2342584965 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f919401b5b0 con 0x7f91a4106a20 2023-07-20T01:21:05.026 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.267+0000 7f91a9716700 1 -- 172.21.15.60:0/2342584965 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91a410a0f0 con 0x7f91a4106a20 2023-07-20T01:21:05.026 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.267+0000 7f918bfff700 1 --2- 172.21.15.60:0/2342584965 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7f918c038760 0x7f918c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:05.027 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.267+0000 7f918bfff700 1 -- 172.21.15.60:0/2342584965 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f919404c5c0 con 0x7f91a4106a20 2023-07-20T01:21:05.027 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.267+0000 7f91a27fc700 1 -- 172.21.15.60:0/2342584965 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7f918c038760 msgr2=0x7f918c03ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/1628989585 2023-07-20T01:21:05.027 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.267+0000 7f91a27fc700 1 --2- 172.21.15.60:0/2342584965 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7f918c038760 0x7f918c03ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T01:21:05.027 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.272+0000 7f918bfff700 1 -- 172.21.15.60:0/2342584965 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9194023590 con 0x7f91a4106a20 2023-07-20T01:21:05.027 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.468+0000 7f91a27fc700 1 -- 172.21.15.60:0/2342584965 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7f918c038760 msgr2=0x7f918c03ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/1628989585 2023-07-20T01:21:05.027 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.468+0000 7f91a27fc700 1 --2- 172.21.15.60:0/2342584965 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7f918c038760 0x7f918c03ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-20T01:21:05.027 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.735+0000 7f91a9716700 1 -- 172.21.15.60:0/2342584965 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f91a419d560 con 0x7f91a4106a20 2023-07-20T01:21:05.028 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.736+0000 7f918bfff700 1 -- 172.21.15.60:0/2342584965 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f919401f020 con 0x7f91a4106a20 2023-07-20T01:21:05.028 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.738+0000 7f91a9716700 1 -- 172.21.15.60:0/2342584965 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7f918c038760 msgr2=0x7f918c03ac20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T01:21:05.028 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.738+0000 7f91a9716700 1 --2- 172.21.15.60:0/2342584965 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7f918c038760 0x7f918c03ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:05.028 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.738+0000 7f91a9716700 1 -- 172.21.15.60:0/2342584965 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91a4106a20 msgr2=0x7f91a4199ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:05.028 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.738+0000 7f91a9716700 1 --2- 172.21.15.60:0/2342584965 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91a4106a20 0x7f91a4199ba0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9194011dc0 tx=0x7f9194011ea0 comp rx=0 tx=0).stop 2023-07-20T01:21:05.028 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.738+0000 7f91a9716700 1 -- 172.21.15.60:0/2342584965 shutdown_connections 2023-07-20T01:21:05.028 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.738+0000 7f91a9716700 1 --2- 172.21.15.60:0/2342584965 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7f918c038760 0x7f918c03ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:05.029 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.738+0000 7f91a9716700 1 --2- 172.21.15.60:0/2342584965 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91a4106a20 0x7f91a4199ba0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:05.029 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.738+0000 7f91a9716700 1 -- 172.21.15.60:0/2342584965 >> 172.21.15.60:0/2342584965 conn(0x7f91a4100490 msgr2=0x7f91a4101250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:05.029 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.738+0000 7f91a9716700 1 -- 172.21.15.60:0/2342584965 shutdown_connections 2023-07-20T01:21:05.029 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:04.738+0000 7f91a9716700 1 -- 172.21.15.60:0/2342584965 wait complete. 2023-07-20T01:21:05.029 INFO:teuthology.orchestra.run.smithi060.stdout:Waiting for the mgr to restart... 2023-07-20T01:21:05.029 INFO:teuthology.orchestra.run.smithi060.stdout:Waiting for mgr epoch 5... 2023-07-20T01:21:05.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:05 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/2342584965' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-20T01:21:05.680 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:05 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:05.428+0000 7f73f279c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T01:21:06.429 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:06 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:06.113+0000 7f73f279c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T01:21:06.430 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:06 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:06.305+0000 7f73f279c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T01:21:07.579 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:07 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:07.268+0000 7f73f279c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T01:21:07.579 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:07 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:07.378+0000 7f73f279c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T01:21:08.180 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:07 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:07.596+0000 7f73f279c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T01:21:08.180 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:07 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:07.945+0000 7f73f279c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T01:21:08.679 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:08 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:08.223+0000 7f73f279c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T01:21:08.679 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:08 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:08.404+0000 7f73f279c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T01:21:09.179 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:08 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:08.733+0000 7f73f279c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T01:21:09.929 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:09 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:09.556+0000 7f73f279c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T01:21:09.929 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:09 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:09.713+0000 7f73f279c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T01:21:10.344 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:09 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:09.955+0000 7f73f279c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T01:21:10.344 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:10 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:10.091+0000 7f73f279c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T01:21:10.679 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:10 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:10.361+0000 7f73f279c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T01:21:11.679 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:11 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:11.404+0000 7f73f279c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T01:21:12.591 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:12 smithi060 ceph-mon[96524]: Active manager daemon x restarted 2023-07-20T01:21:12.592 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:12 smithi060 ceph-mon[96524]: Activating manager daemon x 2023-07-20T01:21:12.592 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:12 smithi060 ceph-mon[96524]: osdmap e2: 0 total, 0 up, 0 in 2023-07-20T01:21:12.592 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:12 smithi060 ceph-mon[96524]: mgrmap e6: x(active, starting, since 0.00333782s) 2023-07-20T01:21:12.592 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:12 smithi060 ceph-mon[96524]: from='mgr.14120 172.21.15.60:0/4194926595' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:21:12.593 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:12 smithi060 ceph-mon[96524]: from='mgr.14120 172.21.15.60:0/4194926595' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-20T01:21:12.593 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:12 smithi060 ceph-mon[96524]: from='mgr.14120 172.21.15.60:0/4194926595' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-20T01:21:12.593 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:12 smithi060 ceph-mon[96524]: from='mgr.14120 172.21.15.60:0/4194926595' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-20T01:21:12.593 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:12 smithi060 ceph-mon[96524]: from='mgr.14120 172.21.15.60:0/4194926595' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-20T01:21:12.593 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:12 smithi060 ceph-mon[96524]: Manager daemon x is now available 2023-07-20T01:21:12.594 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:12 smithi060 ceph-mon[96524]: from='mgr.14120 172.21.15.60:0/4194926595' entity='mgr.x' 2023-07-20T01:21:12.594 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:12 smithi060 ceph-mon[96524]: from='mgr.14120 172.21.15.60:0/4194926595' entity='mgr.x' 2023-07-20T01:21:12.594 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:12 smithi060 ceph-mon[96524]: from='mgr.14120 172.21.15.60:0/4194926595' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:21:12.594 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:12 smithi060 ceph-mon[96524]: from='mgr.14120 172.21.15.60:0/4194926595' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:21:12.594 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:12 smithi060 ceph-mon[96524]: from='mgr.14120 172.21.15.60:0/4194926595' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:21:12.595 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:12 smithi060 ceph-mon[96524]: from='mgr.14120 172.21.15.60:0/4194926595' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-20T01:21:12.595 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:12 smithi060 ceph-mon[96524]: from='mgr.14120 172.21.15.60:0/4194926595' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-20T01:21:12.780 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout { 2023-07-20T01:21:12.781 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-07-20T01:21:12.781 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-20T01:21:12.781 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout } 2023-07-20T01:21:12.781 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.549+0000 7fe33d1bf700 1 Processor -- start 2023-07-20T01:21:12.781 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.549+0000 7fe33d1bf700 1 -- start start 2023-07-20T01:21:12.782 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.549+0000 7fe33d1bf700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe33807ad30 0x7fe338079190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:12.782 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.549+0000 7fe33d1bf700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe338079760 con 0x7fe33807ad30 2023-07-20T01:21:12.782 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.549+0000 7fe336d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe33807ad30 0x7fe338079190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:12.782 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.550+0000 7fe336d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe33807ad30 0x7fe338079190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41868/0 (socket says 172.21.15.60:41868) 2023-07-20T01:21:12.782 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.550+0000 7fe336d9d700 1 -- 172.21.15.60:0/4290368384 learned_addr learned my addr 172.21.15.60:0/4290368384 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:12.782 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.550+0000 7fe336d9d700 1 -- 172.21.15.60:0/4290368384 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3380798a0 con 0x7fe33807ad30 2023-07-20T01:21:12.782 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.550+0000 7fe336d9d700 1 --2- 172.21.15.60:0/4290368384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe33807ad30 0x7fe338079190 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe32c0098a0 tx=0x7fe32c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70c59620b4f1c2d0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:12.783 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.550+0000 7fe335d9b700 1 -- 172.21.15.60:0/4290368384 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe32c004160 con 0x7fe33807ad30 2023-07-20T01:21:12.783 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.550+0000 7fe335d9b700 1 -- 172.21.15.60:0/4290368384 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fe32c0042c0 con 0x7fe33807ad30 2023-07-20T01:21:12.783 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.550+0000 7fe335d9b700 1 -- 172.21.15.60:0/4290368384 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe32c011940 con 0x7fe33807ad30 2023-07-20T01:21:12.783 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.551+0000 7fe33d1bf700 1 -- 172.21.15.60:0/4290368384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe33807ad30 msgr2=0x7fe338079190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:12.783 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.551+0000 7fe33d1bf700 1 --2- 172.21.15.60:0/4290368384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe33807ad30 0x7fe338079190 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe32c0098a0 tx=0x7fe32c009bb0 comp rx=0 tx=0).stop 2023-07-20T01:21:12.783 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.551+0000 7fe33d1bf700 1 -- 172.21.15.60:0/4290368384 shutdown_connections 2023-07-20T01:21:12.783 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.551+0000 7fe33d1bf700 1 --2- 172.21.15.60:0/4290368384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe33807ad30 0x7fe338079190 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:12.784 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.551+0000 7fe33d1bf700 1 -- 172.21.15.60:0/4290368384 >> 172.21.15.60:0/4290368384 conn(0x7fe338101580 msgr2=0x7fe3381039a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:12.784 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.551+0000 7fe33d1bf700 1 -- 172.21.15.60:0/4290368384 shutdown_connections 2023-07-20T01:21:12.784 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.551+0000 7fe33d1bf700 1 -- 172.21.15.60:0/4290368384 wait complete. 2023-07-20T01:21:12.784 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.552+0000 7fe33d1bf700 1 Processor -- start 2023-07-20T01:21:12.784 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.552+0000 7fe33d1bf700 1 -- start start 2023-07-20T01:21:12.784 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.552+0000 7fe33d1bf700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe33807ad30 0x7fe33819be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:12.785 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.552+0000 7fe33d1bf700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33819c3d0 con 0x7fe33807ad30 2023-07-20T01:21:12.785 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.552+0000 7fe336d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe33807ad30 0x7fe33819be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:12.785 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.552+0000 7fe336d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe33807ad30 0x7fe33819be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41870/0 (socket says 172.21.15.60:41870) 2023-07-20T01:21:12.785 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.552+0000 7fe336d9d700 1 -- 172.21.15.60:0/177943640 learned_addr learned my addr 172.21.15.60:0/177943640 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:12.785 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.552+0000 7fe336d9d700 1 -- 172.21.15.60:0/177943640 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe32c009550 con 0x7fe33807ad30 2023-07-20T01:21:12.785 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.553+0000 7fe336d9d700 1 --2- 172.21.15.60:0/177943640 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe33807ad30 0x7fe33819be90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe32c000c00 tx=0x7fe32c011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:12.785 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.553+0000 7fe31ffff700 1 -- 172.21.15.60:0/177943640 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe32c0036a0 con 0x7fe33807ad30 2023-07-20T01:21:12.785 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.553+0000 7fe31ffff700 1 -- 172.21.15.60:0/177943640 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fe32c003c70 con 0x7fe33807ad30 2023-07-20T01:21:12.786 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.553+0000 7fe33d1bf700 1 -- 172.21.15.60:0/177943640 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe33819c5d0 con 0x7fe33807ad30 2023-07-20T01:21:12.786 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.553+0000 7fe31ffff700 1 -- 172.21.15.60:0/177943640 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe32c0193f0 con 0x7fe33807ad30 2023-07-20T01:21:12.786 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.553+0000 7fe33d1bf700 1 -- 172.21.15.60:0/177943640 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe33819c9f0 con 0x7fe33807ad30 2023-07-20T01:21:12.786 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.554+0000 7fe31ffff700 1 -- 172.21.15.60:0/177943640 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fe32c019670 con 0x7fe33807ad30 2023-07-20T01:21:12.786 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.554+0000 7fe31ffff700 1 --2- 172.21.15.60:0/177943640 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7fe320038710 0x7fe32003abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:12.786 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.554+0000 7fe31ffff700 1 -- 172.21.15.60:0/177943640 --> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe32003b2e0 con 0x7fe320038710 2023-07-20T01:21:12.786 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.554+0000 7fe31ffff700 1 -- 172.21.15.60:0/177943640 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe32c04c570 con 0x7fe33807ad30 2023-07-20T01:21:12.787 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.554+0000 7fe33659c700 1 -- 172.21.15.60:0/177943640 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7fe320038710 msgr2=0x7fe32003abd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/1628989585 2023-07-20T01:21:12.787 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.554+0000 7fe33659c700 1 --2- 172.21.15.60:0/177943640 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7fe320038710 0x7fe32003abd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T01:21:12.787 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.755+0000 7fe33659c700 1 -- 172.21.15.60:0/177943640 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7fe320038710 msgr2=0x7fe32003abd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/1628989585 2023-07-20T01:21:12.787 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:05.755+0000 7fe33659c700 1 --2- 172.21.15.60:0/177943640 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7fe320038710 0x7fe32003abd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-20T01:21:12.787 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:06.156+0000 7fe33659c700 1 -- 172.21.15.60:0/177943640 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7fe320038710 msgr2=0x7fe32003abd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/1628989585 2023-07-20T01:21:12.787 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:06.156+0000 7fe33659c700 1 --2- 172.21.15.60:0/177943640 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7fe320038710 0x7fe32003abd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-20T01:21:12.787 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:06.957+0000 7fe33659c700 1 -- 172.21.15.60:0/177943640 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7fe320038710 msgr2=0x7fe32003abd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/1628989585 2023-07-20T01:21:12.788 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:06.957+0000 7fe33659c700 1 --2- 172.21.15.60:0/177943640 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7fe320038710 0x7fe32003abd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-20T01:21:12.788 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:08.558+0000 7fe33659c700 1 -- 172.21.15.60:0/177943640 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7fe320038710 msgr2=0x7fe32003abd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/1628989585 2023-07-20T01:21:12.788 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:08.558+0000 7fe33659c700 1 --2- 172.21.15.60:0/177943640 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7fe320038710 0x7fe32003abd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-07-20T01:21:12.788 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:11.419+0000 7fe31ffff700 1 -- 172.21.15.60:0/177943640 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mgrmap(e 6) v1 ==== 45250+0+0 (secure 0 0 0) 0x7fe32c018c70 con 0x7fe33807ad30 2023-07-20T01:21:12.788 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:11.419+0000 7fe31ffff700 1 -- 172.21.15.60:0/177943640 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7fe320038710 msgr2=0x7fe32003abd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T01:21:12.788 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:11.419+0000 7fe31ffff700 1 --2- 172.21.15.60:0/177943640 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7fe320038710 0x7fe32003abd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:12.788 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:12.420+0000 7fe31ffff700 1 -- 172.21.15.60:0/177943640 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7fe32c04e0a0 con 0x7fe33807ad30 2023-07-20T01:21:12.789 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:12.420+0000 7fe31ffff700 1 --2- 172.21.15.60:0/177943640 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7fe32003bff0 0x7fe32003e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:12.789 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:12.420+0000 7fe31ffff700 1 -- 172.21.15.60:0/177943640 --> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe32003b2e0 con 0x7fe32003bff0 2023-07-20T01:21:12.789 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:12.420+0000 7fe33659c700 1 --2- 172.21.15.60:0/177943640 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7fe32003bff0 0x7fe32003e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:12.789 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:12.421+0000 7fe33659c700 1 --2- 172.21.15.60:0/177943640 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7fe32003bff0 0x7fe32003e3e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe328003a10 tx=0x7fe328009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:12.789 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:12.422+0000 7fe31ffff700 1 -- 172.21.15.60:0/177943640 <== mgr.14120 v2:172.21.15.60:6800/712953724 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fe32003b2e0 con 0x7fe32003bff0 2023-07-20T01:21:12.789 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:12.433+0000 7fe33d1bf700 1 -- 172.21.15.60:0/177943640 --> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fe33819d1d0 con 0x7fe32003bff0 2023-07-20T01:21:12.789 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:12.433+0000 7fe31ffff700 1 -- 172.21.15.60:0/177943640 <== mgr.14120 v2:172.21.15.60:6800/712953724 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fe33819d1d0 con 0x7fe32003bff0 2023-07-20T01:21:12.789 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:12.433+0000 7fe33d1bf700 1 -- 172.21.15.60:0/177943640 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7fe32003bff0 msgr2=0x7fe32003e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:12.790 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:12.433+0000 7fe33d1bf700 1 --2- 172.21.15.60:0/177943640 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7fe32003bff0 0x7fe32003e3e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe328003a10 tx=0x7fe328009290 comp rx=0 tx=0).stop 2023-07-20T01:21:12.790 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:12.433+0000 7fe33d1bf700 1 -- 172.21.15.60:0/177943640 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe33807ad30 msgr2=0x7fe33819be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:12.790 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:12.433+0000 7fe33d1bf700 1 --2- 172.21.15.60:0/177943640 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe33807ad30 0x7fe33819be90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe32c000c00 tx=0x7fe32c011e70 comp rx=0 tx=0).stop 2023-07-20T01:21:12.790 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:12.433+0000 7fe33d1bf700 1 -- 172.21.15.60:0/177943640 shutdown_connections 2023-07-20T01:21:12.790 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:12.433+0000 7fe33d1bf700 1 --2- 172.21.15.60:0/177943640 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7fe32003bff0 0x7fe32003e3e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:12.790 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:12.433+0000 7fe33d1bf700 1 --2- 172.21.15.60:0/177943640 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe33807ad30 0x7fe33819be90 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:12.790 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:12.433+0000 7fe33d1bf700 1 --2- 172.21.15.60:0/177943640 >> [v2:172.21.15.60:6800/1628989585,v1:172.21.15.60:6801/1628989585] conn(0x7fe320038710 0x7fe32003abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:12.791 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:12.433+0000 7fe33d1bf700 1 -- 172.21.15.60:0/177943640 >> 172.21.15.60:0/177943640 conn(0x7fe338101580 msgr2=0x7fe338102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:12.791 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:12.433+0000 7fe33d1bf700 1 -- 172.21.15.60:0/177943640 shutdown_connections 2023-07-20T01:21:12.791 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:12.433+0000 7fe33d1bf700 1 -- 172.21.15.60:0/177943640 wait complete. 2023-07-20T01:21:12.791 INFO:teuthology.orchestra.run.smithi060.stdout:mgr epoch 5 is available 2023-07-20T01:21:12.791 INFO:teuthology.orchestra.run.smithi060.stdout:Setting orchestrator backend to cephadm... 2023-07-20T01:21:13.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:13 smithi060 ceph-mon[96524]: Found migration_current of "None". Setting to last migration. 2023-07-20T01:21:13.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:13 smithi060 ceph-mon[96524]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-20T01:21:13.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:13 smithi060 ceph-mon[96524]: mgrmap e7: x(active, since 1.00519s) 2023-07-20T01:21:13.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:13 smithi060 ceph-mon[96524]: from='mgr.14120 172.21.15.60:0/4194926595' entity='mgr.x' 2023-07-20T01:21:13.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:13 smithi060 ceph-mon[96524]: from='mgr.14120 172.21.15.60:0/4194926595' entity='mgr.x' 2023-07-20T01:21:14.139 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.339+0000 7fc90e048700 1 Processor -- start 2023-07-20T01:21:14.139 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.339+0000 7fc90e048700 1 -- start start 2023-07-20T01:21:14.139 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.339+0000 7fc90e048700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc908104e70 0x7fc908105290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:14.139 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.339+0000 7fc90e048700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9081057d0 con 0x7fc908104e70 2023-07-20T01:21:14.140 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.339+0000 7fc9077fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc908104e70 0x7fc908105290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:14.140 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.339+0000 7fc9077fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc908104e70 0x7fc908105290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41914/0 (socket says 172.21.15.60:41914) 2023-07-20T01:21:14.140 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.339+0000 7fc9077fe700 1 -- 172.21.15.60:0/312702945 learned_addr learned my addr 172.21.15.60:0/312702945 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:14.140 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.340+0000 7fc9077fe700 1 -- 172.21.15.60:0/312702945 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc908105910 con 0x7fc908104e70 2023-07-20T01:21:14.140 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.340+0000 7fc9077fe700 1 --2- 172.21.15.60:0/312702945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc908104e70 0x7fc908105290 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc8f80098a0 tx=0x7fc8f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=797873ea40244621 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:14.140 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.340+0000 7fc9067fc700 1 -- 172.21.15.60:0/312702945 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc8f8004160 con 0x7fc908104e70 2023-07-20T01:21:14.140 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.340+0000 7fc9067fc700 1 -- 172.21.15.60:0/312702945 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fc8f80042c0 con 0x7fc908104e70 2023-07-20T01:21:14.141 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.340+0000 7fc9067fc700 1 -- 172.21.15.60:0/312702945 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc8f8011940 con 0x7fc908104e70 2023-07-20T01:21:14.141 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.341+0000 7fc90e048700 1 -- 172.21.15.60:0/312702945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc908104e70 msgr2=0x7fc908105290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:14.141 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.341+0000 7fc90e048700 1 --2- 172.21.15.60:0/312702945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc908104e70 0x7fc908105290 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc8f80098a0 tx=0x7fc8f8009bb0 comp rx=0 tx=0).stop 2023-07-20T01:21:14.141 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.341+0000 7fc90e048700 1 -- 172.21.15.60:0/312702945 shutdown_connections 2023-07-20T01:21:14.141 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.341+0000 7fc90e048700 1 --2- 172.21.15.60:0/312702945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc908104e70 0x7fc908105290 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:14.141 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.341+0000 7fc90e048700 1 -- 172.21.15.60:0/312702945 >> 172.21.15.60:0/312702945 conn(0x7fc9081004a0 msgr2=0x7fc9081028c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:14.141 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.341+0000 7fc90e048700 1 -- 172.21.15.60:0/312702945 shutdown_connections 2023-07-20T01:21:14.141 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.341+0000 7fc90e048700 1 -- 172.21.15.60:0/312702945 wait complete. 2023-07-20T01:21:14.142 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.342+0000 7fc90e048700 1 Processor -- start 2023-07-20T01:21:14.142 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.342+0000 7fc90e048700 1 -- start start 2023-07-20T01:21:14.142 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.342+0000 7fc90e048700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc908104e70 0x7fc908073ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:14.142 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.342+0000 7fc90e048700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90819c9b0 con 0x7fc908104e70 2023-07-20T01:21:14.142 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.342+0000 7fc9077fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc908104e70 0x7fc908073ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:14.142 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.342+0000 7fc9077fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc908104e70 0x7fc908073ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41916/0 (socket says 172.21.15.60:41916) 2023-07-20T01:21:14.142 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.342+0000 7fc9077fe700 1 -- 172.21.15.60:0/1471437689 learned_addr learned my addr 172.21.15.60:0/1471437689 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:14.143 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.343+0000 7fc9077fe700 1 -- 172.21.15.60:0/1471437689 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f8009550 con 0x7fc908104e70 2023-07-20T01:21:14.143 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.343+0000 7fc9077fe700 1 --2- 172.21.15.60:0/1471437689 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc908104e70 0x7fc908073ba0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8006af0 tx=0x7fc8f8011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:14.143 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.343+0000 7fc904ff9700 1 -- 172.21.15.60:0/1471437689 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc8f8003bf0 con 0x7fc908104e70 2023-07-20T01:21:14.143 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.343+0000 7fc904ff9700 1 -- 172.21.15.60:0/1471437689 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fc8f801b8b0 con 0x7fc908104e70 2023-07-20T01:21:14.143 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.343+0000 7fc90e048700 1 -- 172.21.15.60:0/1471437689 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc90819caf0 con 0x7fc908104e70 2023-07-20T01:21:14.143 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.343+0000 7fc904ff9700 1 -- 172.21.15.60:0/1471437689 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc8f80197e0 con 0x7fc908104e70 2023-07-20T01:21:14.143 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.343+0000 7fc90e048700 1 -- 172.21.15.60:0/1471437689 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc90819cf10 con 0x7fc908104e70 2023-07-20T01:21:14.143 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.344+0000 7fc904ff9700 1 -- 172.21.15.60:0/1471437689 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7fc8f8019a50 con 0x7fc908104e70 2023-07-20T01:21:14.144 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.344+0000 7fc904ff9700 1 --2- 172.21.15.60:0/1471437689 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7fc8f0038640 0x7fc8f003ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:14.144 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.344+0000 7fc904ff9700 1 -- 172.21.15.60:0/1471437689 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc8f804c860 con 0x7fc908104e70 2023-07-20T01:21:14.144 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.344+0000 7fc906ffd700 1 --2- 172.21.15.60:0/1471437689 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7fc8f0038640 0x7fc8f003ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:14.144 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.344+0000 7fc90e048700 1 -- 172.21.15.60:0/1471437689 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc908061da0 con 0x7fc908104e70 2023-07-20T01:21:14.144 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.345+0000 7fc906ffd700 1 --2- 172.21.15.60:0/1471437689 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7fc8f0038640 0x7fc8f003ab00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc006fd0 tx=0x7fc8fc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:14.144 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.349+0000 7fc904ff9700 1 -- 172.21.15.60:0/1471437689 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc8f80189a0 con 0x7fc908104e70 2023-07-20T01:21:14.144 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.734+0000 7fc90e048700 1 -- 172.21.15.60:0/1471437689 --> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fc908102780 con 0x7fc8f0038640 2023-07-20T01:21:14.144 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.742+0000 7fc904ff9700 1 -- 172.21.15.60:0/1471437689 <== mgr.14120 v2:172.21.15.60:6800/712953724 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc908102780 con 0x7fc8f0038640 2023-07-20T01:21:14.145 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.744+0000 7fc90e048700 1 -- 172.21.15.60:0/1471437689 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7fc8f0038640 msgr2=0x7fc8f003ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:14.145 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.744+0000 7fc90e048700 1 --2- 172.21.15.60:0/1471437689 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7fc8f0038640 0x7fc8f003ab00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc006fd0 tx=0x7fc8fc006e40 comp rx=0 tx=0).stop 2023-07-20T01:21:14.145 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.744+0000 7fc90e048700 1 -- 172.21.15.60:0/1471437689 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc908104e70 msgr2=0x7fc908073ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:14.145 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.744+0000 7fc90e048700 1 --2- 172.21.15.60:0/1471437689 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc908104e70 0x7fc908073ba0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8006af0 tx=0x7fc8f8011a30 comp rx=0 tx=0).stop 2023-07-20T01:21:14.145 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.745+0000 7fc90e048700 1 -- 172.21.15.60:0/1471437689 shutdown_connections 2023-07-20T01:21:14.145 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.745+0000 7fc90e048700 1 --2- 172.21.15.60:0/1471437689 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7fc8f0038640 0x7fc8f003ab00 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:14.145 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.745+0000 7fc90e048700 1 --2- 172.21.15.60:0/1471437689 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc908104e70 0x7fc908073ba0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:14.146 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.745+0000 7fc90e048700 1 -- 172.21.15.60:0/1471437689 >> 172.21.15.60:0/1471437689 conn(0x7fc9081004a0 msgr2=0x7fc908101b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:14.146 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.745+0000 7fc90e048700 1 -- 172.21.15.60:0/1471437689 shutdown_connections 2023-07-20T01:21:14.146 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:13.745+0000 7fc90e048700 1 -- 172.21.15.60:0/1471437689 wait complete. 2023-07-20T01:21:14.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:14 smithi060 ceph-mon[96524]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-20T01:21:14.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:14 smithi060 ceph-mon[96524]: from='mgr.14120 172.21.15.60:0/4194926595' entity='mgr.x' 2023-07-20T01:21:14.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:14 smithi060 ceph-mon[96524]: from='mgr.14120 172.21.15.60:0/4194926595' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:21:15.381 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout value unchanged 2023-07-20T01:21:15.388 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.627+0000 7f7379b41700 1 Processor -- start 2023-07-20T01:21:15.389 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.627+0000 7f7379b41700 1 -- start start 2023-07-20T01:21:15.389 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.627+0000 7f7379b41700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73741042b0 0x7f73741046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:15.389 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.627+0000 7f7379b41700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7374104ca0 con 0x7f73741042b0 2023-07-20T01:21:15.389 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.627+0000 7f73737fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73741042b0 0x7f73741046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:15.389 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.627+0000 7f73737fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73741042b0 0x7f73741046d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41928/0 (socket says 172.21.15.60:41928) 2023-07-20T01:21:15.390 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.627+0000 7f73737fe700 1 -- 172.21.15.60:0/1294516656 learned_addr learned my addr 172.21.15.60:0/1294516656 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:15.390 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.627+0000 7f73737fe700 1 -- 172.21.15.60:0/1294516656 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7374107240 con 0x7f73741042b0 2023-07-20T01:21:15.390 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.627+0000 7f73737fe700 1 --2- 172.21.15.60:0/1294516656 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73741042b0 0x7f73741046d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f73640098a0 tx=0x7f7364009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97c3996b760ec2ca server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:15.390 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.628+0000 7f73727fc700 1 -- 172.21.15.60:0/1294516656 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7364004160 con 0x7f73741042b0 2023-07-20T01:21:15.390 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.628+0000 7f73727fc700 1 -- 172.21.15.60:0/1294516656 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f73640042c0 con 0x7f73741042b0 2023-07-20T01:21:15.390 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.628+0000 7f73727fc700 1 -- 172.21.15.60:0/1294516656 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7364011940 con 0x7f73741042b0 2023-07-20T01:21:15.390 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.628+0000 7f7379b41700 1 -- 172.21.15.60:0/1294516656 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73741042b0 msgr2=0x7f73741046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:15.391 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.628+0000 7f7379b41700 1 --2- 172.21.15.60:0/1294516656 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73741042b0 0x7f73741046d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f73640098a0 tx=0x7f7364009bb0 comp rx=0 tx=0).stop 2023-07-20T01:21:15.391 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.628+0000 7f7379b41700 1 -- 172.21.15.60:0/1294516656 shutdown_connections 2023-07-20T01:21:15.391 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.628+0000 7f7379b41700 1 --2- 172.21.15.60:0/1294516656 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73741042b0 0x7f73741046d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:15.391 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.628+0000 7f7379b41700 1 -- 172.21.15.60:0/1294516656 >> 172.21.15.60:0/1294516656 conn(0x7f73740ffc00 msgr2=0x7f7374102020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:15.391 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.628+0000 7f7379b41700 1 -- 172.21.15.60:0/1294516656 shutdown_connections 2023-07-20T01:21:15.391 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.628+0000 7f7379b41700 1 -- 172.21.15.60:0/1294516656 wait complete. 2023-07-20T01:21:15.391 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.629+0000 7f7379b41700 1 Processor -- start 2023-07-20T01:21:15.392 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.629+0000 7f7379b41700 1 -- start start 2023-07-20T01:21:15.392 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.629+0000 7f7379b41700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73741042b0 0x7f7374106ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:15.392 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.629+0000 7f7379b41700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73741070e0 con 0x7f73741042b0 2023-07-20T01:21:15.392 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.629+0000 7f73737fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73741042b0 0x7f7374106ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:15.392 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.630+0000 7f73737fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73741042b0 0x7f7374106ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41930/0 (socket says 172.21.15.60:41930) 2023-07-20T01:21:15.392 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.630+0000 7f73737fe700 1 -- 172.21.15.60:0/2012886268 learned_addr learned my addr 172.21.15.60:0/2012886268 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:15.392 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.630+0000 7f73737fe700 1 -- 172.21.15.60:0/2012886268 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7364009550 con 0x7f73741042b0 2023-07-20T01:21:15.392 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.630+0000 7f73737fe700 1 --2- 172.21.15.60:0/2012886268 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73741042b0 0x7f7374106ba0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7364000c00 tx=0x7f7364011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:15.393 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.630+0000 7f7370ff9700 1 -- 172.21.15.60:0/2012886268 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f73640036a0 con 0x7f73741042b0 2023-07-20T01:21:15.393 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.630+0000 7f7379b41700 1 -- 172.21.15.60:0/2012886268 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73741051f0 con 0x7f73741042b0 2023-07-20T01:21:15.393 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.630+0000 7f7379b41700 1 -- 172.21.15.60:0/2012886268 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7374105610 con 0x7f73741042b0 2023-07-20T01:21:15.393 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.630+0000 7f7370ff9700 1 -- 172.21.15.60:0/2012886268 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f736401b440 con 0x7f73741042b0 2023-07-20T01:21:15.393 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.630+0000 7f7370ff9700 1 -- 172.21.15.60:0/2012886268 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f73640193f0 con 0x7f73741042b0 2023-07-20T01:21:15.393 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.631+0000 7f7370ff9700 1 -- 172.21.15.60:0/2012886268 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f7364019620 con 0x7f73741042b0 2023-07-20T01:21:15.393 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.631+0000 7f7370ff9700 1 --2- 172.21.15.60:0/2012886268 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f735c038760 0x7f735c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:15.394 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.631+0000 7f7370ff9700 1 -- 172.21.15.60:0/2012886268 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f736404c4f0 con 0x7f73741042b0 2023-07-20T01:21:15.394 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.631+0000 7f7379b41700 1 -- 172.21.15.60:0/2012886268 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7360005320 con 0x7f73741042b0 2023-07-20T01:21:15.394 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.631+0000 7f7372ffd700 1 --2- 172.21.15.60:0/2012886268 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f735c038760 0x7f735c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:15.394 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.632+0000 7f7372ffd700 1 --2- 172.21.15.60:0/2012886268 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f735c038760 0x7f735c03ac20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7368006fd0 tx=0x7f7368006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:15.394 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.637+0000 7f7370ff9700 1 -- 172.21.15.60:0/2012886268 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f736401f020 con 0x7f73741042b0 2023-07-20T01:21:15.394 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.992+0000 7f7379b41700 1 -- 172.21.15.60:0/2012886268 --> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f7360000bf0 con 0x7f735c038760 2023-07-20T01:21:15.394 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.993+0000 7f7370ff9700 1 -- 172.21.15.60:0/2012886268 <== mgr.14120 v2:172.21.15.60:6800/712953724 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f7360000bf0 con 0x7f735c038760 2023-07-20T01:21:15.394 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.996+0000 7f7379b41700 1 -- 172.21.15.60:0/2012886268 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f735c038760 msgr2=0x7f735c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:15.395 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.996+0000 7f7379b41700 1 --2- 172.21.15.60:0/2012886268 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f735c038760 0x7f735c03ac20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7368006fd0 tx=0x7f7368006e40 comp rx=0 tx=0).stop 2023-07-20T01:21:15.395 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.996+0000 7f7379b41700 1 -- 172.21.15.60:0/2012886268 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73741042b0 msgr2=0x7f7374106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:15.395 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.996+0000 7f7379b41700 1 --2- 172.21.15.60:0/2012886268 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73741042b0 0x7f7374106ba0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7364000c00 tx=0x7f7364011e90 comp rx=0 tx=0).stop 2023-07-20T01:21:15.395 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.996+0000 7f7379b41700 1 -- 172.21.15.60:0/2012886268 shutdown_connections 2023-07-20T01:21:15.395 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.996+0000 7f7379b41700 1 --2- 172.21.15.60:0/2012886268 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f735c038760 0x7f735c03ac20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:15.395 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.996+0000 7f7379b41700 1 --2- 172.21.15.60:0/2012886268 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73741042b0 0x7f7374106ba0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:15.395 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.996+0000 7f7379b41700 1 -- 172.21.15.60:0/2012886268 >> 172.21.15.60:0/2012886268 conn(0x7f73740ffc00 msgr2=0x7f7374100990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:15.396 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.996+0000 7f7379b41700 1 -- 172.21.15.60:0/2012886268 shutdown_connections 2023-07-20T01:21:15.396 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:14.996+0000 7f7379b41700 1 -- 172.21.15.60:0/2012886268 wait complete. 2023-07-20T01:21:15.396 INFO:teuthology.orchestra.run.smithi060.stdout:Generating ssh key... 2023-07-20T01:21:15.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:15 smithi060 ceph-mon[96524]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:21:15.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:15 smithi060 ceph-mon[96524]: [20/Jul/2023:01:21:14] ENGINE Bus STARTING 2023-07-20T01:21:15.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:15 smithi060 ceph-mon[96524]: [20/Jul/2023:01:21:14] ENGINE Serving on http://172.21.15.60:8765 2023-07-20T01:21:15.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:15 smithi060 ceph-mon[96524]: mgrmap e8: x(active, since 3s) 2023-07-20T01:21:15.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:15 smithi060 ceph-mon[96524]: from='mgr.14120 172.21.15.60:0/4194926595' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:21:16.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:16 smithi060 ceph-mon[96524]: [20/Jul/2023:01:21:14] ENGINE Serving on https://172.21.15.60:7150 2023-07-20T01:21:16.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:16 smithi060 ceph-mon[96524]: [20/Jul/2023:01:21:14] ENGINE Bus STARTED 2023-07-20T01:21:16.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:16 smithi060 ceph-mon[96524]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:21:16.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:16 smithi060 ceph-mon[96524]: from='mgr.14120 172.21.15.60:0/4194926595' entity='mgr.x' 2023-07-20T01:21:16.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:16 smithi060 ceph-mon[96524]: from='mgr.14120 172.21.15.60:0/4194926595' entity='mgr.x' 2023-07-20T01:21:16.680 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:16 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: Generating public/private rsa key pair. 2023-07-20T01:21:16.680 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:16 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: Your identification has been saved in /tmp/tmp0or2z7vx/key. 2023-07-20T01:21:16.680 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:16 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: Your public key has been saved in /tmp/tmp0or2z7vx/key.pub. 2023-07-20T01:21:16.681 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:16 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: The key fingerprint is: 2023-07-20T01:21:16.681 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:16 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: SHA256:ETP96VJkF0avpFC663m2tN0waXOICEp43nlFWPsTqlk ceph-7daf031e-269b-11ee-9b34-001a4aab830c 2023-07-20T01:21:16.681 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:16 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: The key's randomart image is: 2023-07-20T01:21:16.681 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:16 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: +---[RSA 3072]----+ 2023-07-20T01:21:16.681 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:16 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: | +. o.+. | 2023-07-20T01:21:16.681 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:16 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: | +.=oo.. | 2023-07-20T01:21:16.681 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:16 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: | . ++ooo .| 2023-07-20T01:21:16.682 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:16 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: | . . ++= o | 2023-07-20T01:21:16.682 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:16 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: | . o S .oE + | 2023-07-20T01:21:16.682 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:16 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: | + o o.B.. + | 2023-07-20T01:21:16.682 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:16 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: | o o *.o B .| 2023-07-20T01:21:16.682 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:16 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: | o oo+ * | 2023-07-20T01:21:16.682 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:16 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: | ooo.. .| 2023-07-20T01:21:16.682 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:16 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: +----[SHA256]-----+ 2023-07-20T01:21:16.833 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.849+0000 7f74c317c700 1 Processor -- start 2023-07-20T01:21:16.834 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.849+0000 7f74c317c700 1 -- start start 2023-07-20T01:21:16.834 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.849+0000 7f74c317c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74bc104870 0x7f74bc104c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:16.834 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.849+0000 7f74c317c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc079df0 con 0x7f74bc104870 2023-07-20T01:21:16.834 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.850+0000 7f74c0f18700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74bc104870 0x7f74bc104c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:16.834 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.850+0000 7f74c0f18700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74bc104870 0x7f74bc104c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41934/0 (socket says 172.21.15.60:41934) 2023-07-20T01:21:16.834 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.850+0000 7f74c0f18700 1 -- 172.21.15.60:0/3688851755 learned_addr learned my addr 172.21.15.60:0/3688851755 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:16.835 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.850+0000 7f74c0f18700 1 -- 172.21.15.60:0/3688851755 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74bc1051d0 con 0x7f74bc104870 2023-07-20T01:21:16.835 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.850+0000 7f74c0f18700 1 --2- 172.21.15.60:0/3688851755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74bc104870 0x7f74bc104c90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f74b00098a0 tx=0x7f74b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f7cc227f16d06b3 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:16.835 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.850+0000 7f74bb7fe700 1 -- 172.21.15.60:0/3688851755 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f74b0004160 con 0x7f74bc104870 2023-07-20T01:21:16.835 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.850+0000 7f74bb7fe700 1 -- 172.21.15.60:0/3688851755 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f74b00042c0 con 0x7f74bc104870 2023-07-20T01:21:16.835 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.850+0000 7f74bb7fe700 1 -- 172.21.15.60:0/3688851755 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f74b0011940 con 0x7f74bc104870 2023-07-20T01:21:16.835 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.851+0000 7f74c317c700 1 -- 172.21.15.60:0/3688851755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74bc104870 msgr2=0x7f74bc104c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:16.836 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.851+0000 7f74c317c700 1 --2- 172.21.15.60:0/3688851755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74bc104870 0x7f74bc104c90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f74b00098a0 tx=0x7f74b0009bb0 comp rx=0 tx=0).stop 2023-07-20T01:21:16.836 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.851+0000 7f74c317c700 1 -- 172.21.15.60:0/3688851755 shutdown_connections 2023-07-20T01:21:16.836 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.851+0000 7f74c317c700 1 --2- 172.21.15.60:0/3688851755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74bc104870 0x7f74bc104c90 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:16.836 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.851+0000 7f74c317c700 1 -- 172.21.15.60:0/3688851755 >> 172.21.15.60:0/3688851755 conn(0x7f74bc1004b0 msgr2=0x7f74bc1028d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:16.836 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.851+0000 7f74c317c700 1 -- 172.21.15.60:0/3688851755 shutdown_connections 2023-07-20T01:21:16.836 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.851+0000 7f74c317c700 1 -- 172.21.15.60:0/3688851755 wait complete. 2023-07-20T01:21:16.837 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.852+0000 7f74c317c700 1 Processor -- start 2023-07-20T01:21:16.837 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.852+0000 7f74c317c700 1 -- start start 2023-07-20T01:21:16.837 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.852+0000 7f74c317c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74bc104870 0x7f74bc19be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:16.837 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.852+0000 7f74c317c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc19c370 con 0x7f74bc104870 2023-07-20T01:21:16.837 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.852+0000 7f74c0f18700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74bc104870 0x7f74bc19be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:16.837 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.852+0000 7f74c0f18700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74bc104870 0x7f74bc19be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41936/0 (socket says 172.21.15.60:41936) 2023-07-20T01:21:16.838 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.852+0000 7f74c0f18700 1 -- 172.21.15.60:0/1239638196 learned_addr learned my addr 172.21.15.60:0/1239638196 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:16.838 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.852+0000 7f74c0f18700 1 -- 172.21.15.60:0/1239638196 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b0009550 con 0x7f74bc104870 2023-07-20T01:21:16.838 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.852+0000 7f74c0f18700 1 --2- 172.21.15.60:0/1239638196 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74bc104870 0x7f74bc19be30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f74b0006af0 tx=0x7f74b0011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:16.838 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.852+0000 7f74b9ffb700 1 -- 172.21.15.60:0/1239638196 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f74b0003bf0 con 0x7f74bc104870 2023-07-20T01:21:16.838 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.852+0000 7f74b9ffb700 1 -- 172.21.15.60:0/1239638196 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f74b001b8b0 con 0x7f74bc104870 2023-07-20T01:21:16.838 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.852+0000 7f74c317c700 1 -- 172.21.15.60:0/1239638196 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74bc19c570 con 0x7f74bc104870 2023-07-20T01:21:16.839 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.853+0000 7f74b9ffb700 1 -- 172.21.15.60:0/1239638196 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f74b00197b0 con 0x7f74bc104870 2023-07-20T01:21:16.839 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.853+0000 7f74c317c700 1 -- 172.21.15.60:0/1239638196 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74bc19c990 con 0x7f74bc104870 2023-07-20T01:21:16.839 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.854+0000 7f74b9ffb700 1 -- 172.21.15.60:0/1239638196 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f74b0019a20 con 0x7f74bc104870 2023-07-20T01:21:16.839 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.854+0000 7f74b9ffb700 1 --2- 172.21.15.60:0/1239638196 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f74ac038710 0x7f74ac03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:16.839 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.854+0000 7f74c317c700 1 -- 172.21.15.60:0/1239638196 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74bc195750 con 0x7f74bc104870 2023-07-20T01:21:16.839 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.854+0000 7f74b9ffb700 1 -- 172.21.15.60:0/1239638196 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f74b004cbc0 con 0x7f74bc104870 2023-07-20T01:21:16.840 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.854+0000 7f74bbfff700 1 --2- 172.21.15.60:0/1239638196 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f74ac038710 0x7f74ac03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:16.840 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.854+0000 7f74bbfff700 1 --2- 172.21.15.60:0/1239638196 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f74ac038710 0x7f74ac03abd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f74a8006fd0 tx=0x7f74a8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:16.840 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:15.859+0000 7f74b9ffb700 1 -- 172.21.15.60:0/1239638196 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f74b0018bd0 con 0x7f74bc104870 2023-07-20T01:21:16.840 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:16.217+0000 7f74c317c700 1 -- 172.21.15.60:0/1239638196 --> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f74bc060bb0 con 0x7f74ac038710 2023-07-20T01:21:16.840 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:16.419+0000 7f74b9ffb700 1 -- 172.21.15.60:0/1239638196 <== mgr.14120 v2:172.21.15.60:6800/712953724 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f74bc060bb0 con 0x7f74ac038710 2023-07-20T01:21:16.840 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:16.422+0000 7f74c317c700 1 -- 172.21.15.60:0/1239638196 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f74ac038710 msgr2=0x7f74ac03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:16.841 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:16.422+0000 7f74c317c700 1 --2- 172.21.15.60:0/1239638196 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f74ac038710 0x7f74ac03abd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f74a8006fd0 tx=0x7f74a8006e40 comp rx=0 tx=0).stop 2023-07-20T01:21:16.841 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:16.422+0000 7f74c317c700 1 -- 172.21.15.60:0/1239638196 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74bc104870 msgr2=0x7f74bc19be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:16.841 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:16.422+0000 7f74c317c700 1 --2- 172.21.15.60:0/1239638196 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74bc104870 0x7f74bc19be30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f74b0006af0 tx=0x7f74b0011a30 comp rx=0 tx=0).stop 2023-07-20T01:21:16.841 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:16.422+0000 7f74c317c700 1 -- 172.21.15.60:0/1239638196 shutdown_connections 2023-07-20T01:21:16.841 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:16.422+0000 7f74c317c700 1 --2- 172.21.15.60:0/1239638196 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f74ac038710 0x7f74ac03abd0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:16.842 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:16.422+0000 7f74c317c700 1 --2- 172.21.15.60:0/1239638196 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74bc104870 0x7f74bc19be30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:16.842 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:16.422+0000 7f74c317c700 1 -- 172.21.15.60:0/1239638196 >> 172.21.15.60:0/1239638196 conn(0x7f74bc1004b0 msgr2=0x7f74bc190430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:16.842 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:16.422+0000 7f74c317c700 1 -- 172.21.15.60:0/1239638196 shutdown_connections 2023-07-20T01:21:16.842 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:16.422+0000 7f74c317c700 1 -- 172.21.15.60:0/1239638196 wait complete. 2023-07-20T01:21:17.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:17 smithi060 ceph-mon[96524]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:21:17.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:17 smithi060 ceph-mon[96524]: Generating ssh key... 2023-07-20T01:21:18.283 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5hIUHf9mSLFFtoqFD3PELkONSJSPyFd15dH1JSzJXBrjCyLmD7+nuw6q+3CxWbmzX5IXq89O79SztAXJkw5TfRRa6jV8sFBLs6Ch0eMSbh30fXWDomymT1L6sDcWMFwKwwE8UHCWN2qsoj6omxciFCUPWYMIHV+xKfc7Hk2DWPNnV31bgE92HjPFK5exGyzrFDYe7fERXdcg1GXylc+7z50WaMYbRTBNGvavOZRPmH5kRVGs3wL1iqpdHuH4IJwkE5UyGAcdghijEXPvTyoVjfXhahnJCrNE/bpeVPmtMBlCFf2HsNzxGAK5ugNCDns2ToOI3FyGc9NFD1ev0vUBH2hgT/Ht9x/9tqKGY33d9/F7PBFmGFB2X0Ky6Vaukp17F+ekERa8B8wuh7Wdcn3ErmLctpMA2CbE5VSm28Rngv9dA9ZEOnESjid82BLCZ/Z34iUGGsCh2TIgK6RwRPLCFmwHj8AcN1qAHQ3UAMM5x5K7NkSUUfvEaqZ4l+UMxWE8= ceph-7daf031e-269b-11ee-9b34-001a4aab830c 2023-07-20T01:21:18.283 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.343+0000 7f944fa11700 1 Processor -- start 2023-07-20T01:21:18.283 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.343+0000 7f944fa11700 1 -- start start 2023-07-20T01:21:18.284 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.343+0000 7f944fa11700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9448105fb0 0x7f94481063d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:18.284 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.343+0000 7f944fa11700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94481069a0 con 0x7f9448105fb0 2023-07-20T01:21:18.284 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.343+0000 7f944d7ad700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9448105fb0 0x7f94481063d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:18.284 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.343+0000 7f944d7ad700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9448105fb0 0x7f94481063d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41940/0 (socket says 172.21.15.60:41940) 2023-07-20T01:21:18.285 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.343+0000 7f944d7ad700 1 -- 172.21.15.60:0/3631987354 learned_addr learned my addr 172.21.15.60:0/3631987354 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:18.285 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.344+0000 7f944d7ad700 1 -- 172.21.15.60:0/3631987354 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94481071b0 con 0x7f9448105fb0 2023-07-20T01:21:18.285 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.344+0000 7f944d7ad700 1 --2- 172.21.15.60:0/3631987354 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9448105fb0 0x7f94481063d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f94440098a0 tx=0x7f9444009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8cb3a16097b6ad0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:18.286 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.344+0000 7f943ffff700 1 -- 172.21.15.60:0/3631987354 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9444004160 con 0x7f9448105fb0 2023-07-20T01:21:18.286 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.344+0000 7f943ffff700 1 -- 172.21.15.60:0/3631987354 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f94440042c0 con 0x7f9448105fb0 2023-07-20T01:21:18.286 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.344+0000 7f943ffff700 1 -- 172.21.15.60:0/3631987354 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9444011940 con 0x7f9448105fb0 2023-07-20T01:21:18.286 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.344+0000 7f944fa11700 1 -- 172.21.15.60:0/3631987354 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9448105fb0 msgr2=0x7f94481063d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:18.287 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.344+0000 7f944fa11700 1 --2- 172.21.15.60:0/3631987354 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9448105fb0 0x7f94481063d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f94440098a0 tx=0x7f9444009bb0 comp rx=0 tx=0).stop 2023-07-20T01:21:18.287 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.345+0000 7f944fa11700 1 -- 172.21.15.60:0/3631987354 shutdown_connections 2023-07-20T01:21:18.287 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.345+0000 7f944fa11700 1 --2- 172.21.15.60:0/3631987354 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9448105fb0 0x7f94481063d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:18.287 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.345+0000 7f944fa11700 1 -- 172.21.15.60:0/3631987354 >> 172.21.15.60:0/3631987354 conn(0x7f9448101590 msgr2=0x7f94481039b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:18.288 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.345+0000 7f944fa11700 1 -- 172.21.15.60:0/3631987354 shutdown_connections 2023-07-20T01:21:18.288 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.345+0000 7f944fa11700 1 -- 172.21.15.60:0/3631987354 wait complete. 2023-07-20T01:21:18.288 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.345+0000 7f944fa11700 1 Processor -- start 2023-07-20T01:21:18.288 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.345+0000 7f944fa11700 1 -- start start 2023-07-20T01:21:18.289 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.345+0000 7f944fa11700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9448105fb0 0x7f9448199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:18.289 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.345+0000 7f944fa11700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f944819a160 con 0x7f9448105fb0 2023-07-20T01:21:18.289 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.346+0000 7f944d7ad700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9448105fb0 0x7f9448199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:18.289 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.346+0000 7f944d7ad700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9448105fb0 0x7f9448199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41942/0 (socket says 172.21.15.60:41942) 2023-07-20T01:21:18.290 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.346+0000 7f944d7ad700 1 -- 172.21.15.60:0/4112443505 learned_addr learned my addr 172.21.15.60:0/4112443505 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:18.290 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.346+0000 7f944d7ad700 1 -- 172.21.15.60:0/4112443505 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9444009550 con 0x7f9448105fb0 2023-07-20T01:21:18.290 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.346+0000 7f944d7ad700 1 --2- 172.21.15.60:0/4112443505 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9448105fb0 0x7f9448199c20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9444000c00 tx=0x7f9444011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:18.290 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.346+0000 7f943e7fc700 1 -- 172.21.15.60:0/4112443505 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f94440036a0 con 0x7f9448105fb0 2023-07-20T01:21:18.291 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.346+0000 7f943e7fc700 1 -- 172.21.15.60:0/4112443505 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f944401b440 con 0x7f9448105fb0 2023-07-20T01:21:18.291 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.346+0000 7f944fa11700 1 -- 172.21.15.60:0/4112443505 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f944819a360 con 0x7f9448105fb0 2023-07-20T01:21:18.291 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.346+0000 7f943e7fc700 1 -- 172.21.15.60:0/4112443505 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f94440193f0 con 0x7f9448105fb0 2023-07-20T01:21:18.292 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.346+0000 7f944fa11700 1 -- 172.21.15.60:0/4112443505 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f944819a780 con 0x7f9448105fb0 2023-07-20T01:21:18.292 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.347+0000 7f943e7fc700 1 -- 172.21.15.60:0/4112443505 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f944401b5b0 con 0x7f9448105fb0 2023-07-20T01:21:18.292 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.347+0000 7f944fa11700 1 -- 172.21.15.60:0/4112443505 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f944810a3e0 con 0x7f9448105fb0 2023-07-20T01:21:18.292 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.348+0000 7f943e7fc700 1 --2- 172.21.15.60:0/4112443505 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f9434038760 0x7f943403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:18.293 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.348+0000 7f943e7fc700 1 -- 172.21.15.60:0/4112443505 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f944404c630 con 0x7f9448105fb0 2023-07-20T01:21:18.293 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.348+0000 7f944cfac700 1 --2- 172.21.15.60:0/4112443505 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f9434038760 0x7f943403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:18.293 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.348+0000 7f944cfac700 1 --2- 172.21.15.60:0/4112443505 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f9434038760 0x7f943403ac20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9438006fd0 tx=0x7f9438006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:18.293 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.352+0000 7f943e7fc700 1 -- 172.21.15.60:0/4112443505 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9444022950 con 0x7f9448105fb0 2023-07-20T01:21:18.294 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.712+0000 7f944fa11700 1 -- 172.21.15.60:0/4112443505 --> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f944819d5a0 con 0x7f9434038760 2023-07-20T01:21:18.294 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.712+0000 7f943e7fc700 1 -- 172.21.15.60:0/4112443505 <== mgr.14120 v2:172.21.15.60:6800/712953724 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f944819d5a0 con 0x7f9434038760 2023-07-20T01:21:18.294 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.715+0000 7f944fa11700 1 -- 172.21.15.60:0/4112443505 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f9434038760 msgr2=0x7f943403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:18.294 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.715+0000 7f944fa11700 1 --2- 172.21.15.60:0/4112443505 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f9434038760 0x7f943403ac20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9438006fd0 tx=0x7f9438006e40 comp rx=0 tx=0).stop 2023-07-20T01:21:18.295 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.715+0000 7f944fa11700 1 -- 172.21.15.60:0/4112443505 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9448105fb0 msgr2=0x7f9448199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:18.295 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.715+0000 7f944fa11700 1 --2- 172.21.15.60:0/4112443505 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9448105fb0 0x7f9448199c20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9444000c00 tx=0x7f9444011e70 comp rx=0 tx=0).stop 2023-07-20T01:21:18.295 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.715+0000 7f944fa11700 1 -- 172.21.15.60:0/4112443505 shutdown_connections 2023-07-20T01:21:18.295 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.715+0000 7f944fa11700 1 --2- 172.21.15.60:0/4112443505 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f9434038760 0x7f943403ac20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:18.296 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.715+0000 7f944fa11700 1 --2- 172.21.15.60:0/4112443505 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9448105fb0 0x7f9448199c20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:18.296 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.715+0000 7f944fa11700 1 -- 172.21.15.60:0/4112443505 >> 172.21.15.60:0/4112443505 conn(0x7f9448101590 msgr2=0x7f9448102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:18.296 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.715+0000 7f944fa11700 1 -- 172.21.15.60:0/4112443505 shutdown_connections 2023-07-20T01:21:18.296 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:17.715+0000 7f944fa11700 1 -- 172.21.15.60:0/4112443505 wait complete. 2023-07-20T01:21:18.297 INFO:teuthology.orchestra.run.smithi060.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-07-20T01:21:18.297 INFO:teuthology.orchestra.run.smithi060.stdout:Adding key to root@localhost authorized_keys... 2023-07-20T01:21:18.297 INFO:teuthology.orchestra.run.smithi060.stdout:Adding host smithi060... 2023-07-20T01:21:18.486 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:18 smithi060 ceph-mon[96524]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:21:19.686 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:19 smithi060 ceph-mon[96524]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi060", "addr": "172.21.15.60", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:21:20.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:20 smithi060 ceph-mon[96524]: Deploying cephadm binary to smithi060 2023-07-20T01:21:22.334 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout Added host 'smithi060' with addr '172.21.15.60' 2023-07-20T01:21:22.334 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.796+0000 7fb737bc6700 1 Processor -- start 2023-07-20T01:21:22.334 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.796+0000 7fb737bc6700 1 -- start start 2023-07-20T01:21:22.334 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.797+0000 7fb737bc6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb730106520 0x7fb730106940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:22.335 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.797+0000 7fb737bc6700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb730106f10 con 0x7fb730106520 2023-07-20T01:21:22.335 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.797+0000 7fb735962700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb730106520 0x7fb730106940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:22.335 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.797+0000 7fb735962700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb730106520 0x7fb730106940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41946/0 (socket says 172.21.15.60:41946) 2023-07-20T01:21:22.335 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.797+0000 7fb735962700 1 -- 172.21.15.60:0/2460119000 learned_addr learned my addr 172.21.15.60:0/2460119000 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:22.335 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.797+0000 7fb735962700 1 -- 172.21.15.60:0/2460119000 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb730107050 con 0x7fb730106520 2023-07-20T01:21:22.335 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.797+0000 7fb735962700 1 --2- 172.21.15.60:0/2460119000 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb730106520 0x7fb730106940 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb7200098a0 tx=0x7fb720009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31a772c18a20e6aa server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:22.335 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.797+0000 7fb734960700 1 -- 172.21.15.60:0/2460119000 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb720004160 con 0x7fb730106520 2023-07-20T01:21:22.335 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.797+0000 7fb734960700 1 -- 172.21.15.60:0/2460119000 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fb7200042c0 con 0x7fb730106520 2023-07-20T01:21:22.336 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.797+0000 7fb734960700 1 -- 172.21.15.60:0/2460119000 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb720011940 con 0x7fb730106520 2023-07-20T01:21:22.336 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.798+0000 7fb737bc6700 1 -- 172.21.15.60:0/2460119000 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb730106520 msgr2=0x7fb730106940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:22.336 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.798+0000 7fb737bc6700 1 --2- 172.21.15.60:0/2460119000 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb730106520 0x7fb730106940 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb7200098a0 tx=0x7fb720009bb0 comp rx=0 tx=0).stop 2023-07-20T01:21:22.336 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.798+0000 7fb737bc6700 1 -- 172.21.15.60:0/2460119000 shutdown_connections 2023-07-20T01:21:22.336 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.798+0000 7fb737bc6700 1 --2- 172.21.15.60:0/2460119000 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb730106520 0x7fb730106940 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:22.336 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.798+0000 7fb737bc6700 1 -- 172.21.15.60:0/2460119000 >> 172.21.15.60:0/2460119000 conn(0x7fb7300ffc00 msgr2=0x7fb730102020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:22.336 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.798+0000 7fb737bc6700 1 -- 172.21.15.60:0/2460119000 shutdown_connections 2023-07-20T01:21:22.337 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.798+0000 7fb737bc6700 1 -- 172.21.15.60:0/2460119000 wait complete. 2023-07-20T01:21:22.337 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.799+0000 7fb737bc6700 1 Processor -- start 2023-07-20T01:21:22.337 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.799+0000 7fb737bc6700 1 -- start start 2023-07-20T01:21:22.337 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.799+0000 7fb737bc6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb730106520 0x7fb73019bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:22.337 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.799+0000 7fb737bc6700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73019c290 con 0x7fb730106520 2023-07-20T01:21:22.337 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.799+0000 7fb735962700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb730106520 0x7fb73019bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:22.337 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.799+0000 7fb735962700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb730106520 0x7fb73019bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41948/0 (socket says 172.21.15.60:41948) 2023-07-20T01:21:22.338 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.799+0000 7fb735962700 1 -- 172.21.15.60:0/213655341 learned_addr learned my addr 172.21.15.60:0/213655341 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:22.338 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.800+0000 7fb735962700 1 -- 172.21.15.60:0/213655341 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb720009550 con 0x7fb730106520 2023-07-20T01:21:22.338 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.800+0000 7fb735962700 1 --2- 172.21.15.60:0/213655341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb730106520 0x7fb73019bd50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb720006af0 tx=0x7fb720011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:22.338 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.800+0000 7fb726ffd700 1 -- 172.21.15.60:0/213655341 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb720003bf0 con 0x7fb730106520 2023-07-20T01:21:22.338 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.800+0000 7fb726ffd700 1 -- 172.21.15.60:0/213655341 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fb72001b8b0 con 0x7fb730106520 2023-07-20T01:21:22.338 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.800+0000 7fb737bc6700 1 -- 172.21.15.60:0/213655341 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb73019c490 con 0x7fb730106520 2023-07-20T01:21:22.338 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.800+0000 7fb726ffd700 1 -- 172.21.15.60:0/213655341 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb720019880 con 0x7fb730106520 2023-07-20T01:21:22.338 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.800+0000 7fb737bc6700 1 -- 172.21.15.60:0/213655341 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb73019c8b0 con 0x7fb730106520 2023-07-20T01:21:22.339 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.801+0000 7fb726ffd700 1 -- 172.21.15.60:0/213655341 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fb72001ba20 con 0x7fb730106520 2023-07-20T01:21:22.339 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.801+0000 7fb737bc6700 1 -- 172.21.15.60:0/213655341 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb730195b80 con 0x7fb730106520 2023-07-20T01:21:22.339 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.801+0000 7fb726ffd700 1 --2- 172.21.15.60:0/213655341 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7fb71c038710 0x7fb71c03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:22.339 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.801+0000 7fb726ffd700 1 -- 172.21.15.60:0/213655341 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb72004c9c0 con 0x7fb730106520 2023-07-20T01:21:22.339 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.801+0000 7fb735161700 1 --2- 172.21.15.60:0/213655341 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7fb71c038710 0x7fb71c03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:22.339 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.802+0000 7fb735161700 1 --2- 172.21.15.60:0/213655341 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7fb71c038710 0x7fb71c03abd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb72c006fd0 tx=0x7fb72c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:22.339 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:18.806+0000 7fb726ffd700 1 -- 172.21.15.60:0/213655341 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb720018530 con 0x7fb730106520 2023-07-20T01:21:22.340 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:19.171+0000 7fb737bc6700 1 -- 172.21.15.60:0/213655341 --> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi060", "addr": "172.21.15.60", "target": ["mon-mgr", ""]}) v1 -- 0x7fb730060bb0 con 0x7fb71c038710 2023-07-20T01:21:22.340 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:21.927+0000 7fb726ffd700 1 -- 172.21.15.60:0/213655341 <== mgr.14120 v2:172.21.15.60:6800/712953724 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fb730060bb0 con 0x7fb71c038710 2023-07-20T01:21:22.340 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:21.930+0000 7fb737bc6700 1 -- 172.21.15.60:0/213655341 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7fb71c038710 msgr2=0x7fb71c03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:22.340 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:21.930+0000 7fb737bc6700 1 --2- 172.21.15.60:0/213655341 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7fb71c038710 0x7fb71c03abd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb72c006fd0 tx=0x7fb72c006e40 comp rx=0 tx=0).stop 2023-07-20T01:21:22.340 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:21.930+0000 7fb737bc6700 1 -- 172.21.15.60:0/213655341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb730106520 msgr2=0x7fb73019bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:22.340 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:21.930+0000 7fb737bc6700 1 --2- 172.21.15.60:0/213655341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb730106520 0x7fb73019bd50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb720006af0 tx=0x7fb720011a30 comp rx=0 tx=0).stop 2023-07-20T01:21:22.340 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:21.930+0000 7fb737bc6700 1 -- 172.21.15.60:0/213655341 shutdown_connections 2023-07-20T01:21:22.341 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:21.930+0000 7fb737bc6700 1 --2- 172.21.15.60:0/213655341 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7fb71c038710 0x7fb71c03abd0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:22.341 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:21.930+0000 7fb737bc6700 1 --2- 172.21.15.60:0/213655341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb730106520 0x7fb73019bd50 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:22.341 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:21.930+0000 7fb737bc6700 1 -- 172.21.15.60:0/213655341 >> 172.21.15.60:0/213655341 conn(0x7fb7300ffc00 msgr2=0x7fb730192cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:22.341 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:21.930+0000 7fb737bc6700 1 -- 172.21.15.60:0/213655341 shutdown_connections 2023-07-20T01:21:22.341 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:21.930+0000 7fb737bc6700 1 -- 172.21.15.60:0/213655341 wait complete. 2023-07-20T01:21:22.341 INFO:teuthology.orchestra.run.smithi060.stdout:Deploying unmanaged mon service... 2023-07-20T01:21:23.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:22 smithi060 ceph-mon[96524]: from='mgr.14120 172.21.15.60:0/4194926595' entity='mgr.x' 2023-07-20T01:21:23.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:22 smithi060 ceph-mon[96524]: Added host smithi060 2023-07-20T01:21:23.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:22 smithi060 ceph-mon[96524]: from='mgr.14120 172.21.15.60:0/4194926595' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:21:23.981 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-07-20T01:21:23.982 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.868+0000 7f89c8061700 1 Processor -- start 2023-07-20T01:21:23.982 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.869+0000 7f89c8061700 1 -- start start 2023-07-20T01:21:23.982 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.869+0000 7f89c8061700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89c0106a90 0x7f89c0108e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:23.982 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.869+0000 7f89c8061700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89c0079dd0 con 0x7f89c0106a90 2023-07-20T01:21:23.982 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.869+0000 7f89c5dfd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89c0106a90 0x7f89c0108e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:23.982 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.869+0000 7f89c5dfd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89c0106a90 0x7f89c0108e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41982/0 (socket says 172.21.15.60:41982) 2023-07-20T01:21:23.983 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.869+0000 7f89c5dfd700 1 -- 172.21.15.60:0/853889510 learned_addr learned my addr 172.21.15.60:0/853889510 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:23.983 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.869+0000 7f89c5dfd700 1 -- 172.21.15.60:0/853889510 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89c01093c0 con 0x7f89c0106a90 2023-07-20T01:21:23.983 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.870+0000 7f89c5dfd700 1 --2- 172.21.15.60:0/853889510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89c0106a90 0x7f89c0108e80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f89b0009e80 tx=0x7f89b000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed61ee9f3b6e1c97 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:23.983 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.870+0000 7f89c4dfb700 1 -- 172.21.15.60:0/853889510 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f89b000e070 con 0x7f89c0106a90 2023-07-20T01:21:23.983 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.870+0000 7f89c4dfb700 1 -- 172.21.15.60:0/853889510 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f89b0003e90 con 0x7f89c0106a90 2023-07-20T01:21:23.983 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.870+0000 7f89c4dfb700 1 -- 172.21.15.60:0/853889510 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f89b0004360 con 0x7f89c0106a90 2023-07-20T01:21:23.983 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.870+0000 7f89c8061700 1 -- 172.21.15.60:0/853889510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89c0106a90 msgr2=0x7f89c0108e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:23.983 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.870+0000 7f89c8061700 1 --2- 172.21.15.60:0/853889510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89c0106a90 0x7f89c0108e80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f89b0009e80 tx=0x7f89b000b330 comp rx=0 tx=0).stop 2023-07-20T01:21:23.984 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.870+0000 7f89c8061700 1 -- 172.21.15.60:0/853889510 shutdown_connections 2023-07-20T01:21:23.984 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.870+0000 7f89c8061700 1 --2- 172.21.15.60:0/853889510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89c0106a90 0x7f89c0108e80 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:23.984 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.870+0000 7f89c8061700 1 -- 172.21.15.60:0/853889510 >> 172.21.15.60:0/853889510 conn(0x7f89c0100480 msgr2=0x7f89c01028a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:23.984 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.870+0000 7f89c8061700 1 -- 172.21.15.60:0/853889510 shutdown_connections 2023-07-20T01:21:23.984 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.871+0000 7f89c8061700 1 -- 172.21.15.60:0/853889510 wait complete. 2023-07-20T01:21:23.984 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.871+0000 7f89c8061700 1 Processor -- start 2023-07-20T01:21:23.984 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.871+0000 7f89c8061700 1 -- start start 2023-07-20T01:21:23.985 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.871+0000 7f89c8061700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89c0106a90 0x7f89c01063f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:23.985 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.871+0000 7f89c8061700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89c0106930 con 0x7f89c0106a90 2023-07-20T01:21:23.985 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.871+0000 7f89c5dfd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89c0106a90 0x7f89c01063f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:23.985 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.871+0000 7f89c5dfd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89c0106a90 0x7f89c01063f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41984/0 (socket says 172.21.15.60:41984) 2023-07-20T01:21:23.985 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.871+0000 7f89c5dfd700 1 -- 172.21.15.60:0/3793435008 learned_addr learned my addr 172.21.15.60:0/3793435008 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:23.985 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.872+0000 7f89c5dfd700 1 -- 172.21.15.60:0/3793435008 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b0009b30 con 0x7f89c0106a90 2023-07-20T01:21:23.986 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.872+0000 7f89c5dfd700 1 --2- 172.21.15.60:0/3793435008 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89c0106a90 0x7f89c01063f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f89b00072a0 tx=0x7f89b0003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:23.986 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.872+0000 7f89beffd700 1 -- 172.21.15.60:0/3793435008 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f89b000e040 con 0x7f89c0106a90 2023-07-20T01:21:23.986 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.872+0000 7f89beffd700 1 -- 172.21.15.60:0/3793435008 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f89b00128a0 con 0x7f89c0106a90 2023-07-20T01:21:23.986 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.872+0000 7f89c8061700 1 -- 172.21.15.60:0/3793435008 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89c0104a40 con 0x7f89c0106a90 2023-07-20T01:21:23.986 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.872+0000 7f89beffd700 1 -- 172.21.15.60:0/3793435008 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f89b001c6b0 con 0x7f89c0106a90 2023-07-20T01:21:23.986 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.872+0000 7f89c8061700 1 -- 172.21.15.60:0/3793435008 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89c0104ec0 con 0x7f89c0106a90 2023-07-20T01:21:23.986 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.873+0000 7f89beffd700 1 -- 172.21.15.60:0/3793435008 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f89b0012430 con 0x7f89c0106a90 2023-07-20T01:21:23.987 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.873+0000 7f89beffd700 1 --2- 172.21.15.60:0/3793435008 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f89a8038740 0x7f89a803ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:23.987 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.873+0000 7f89beffd700 1 -- 172.21.15.60:0/3793435008 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f89b004c630 con 0x7f89c0106a90 2023-07-20T01:21:23.987 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.873+0000 7f89c8061700 1 -- 172.21.15.60:0/3793435008 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89c010a0b0 con 0x7f89c0106a90 2023-07-20T01:21:23.987 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.873+0000 7f89c55fc700 1 --2- 172.21.15.60:0/3793435008 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f89a8038740 0x7f89a803ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:23.987 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.874+0000 7f89c55fc700 1 --2- 172.21.15.60:0/3793435008 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f89a8038740 0x7f89a803ac00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f89b4006fd0 tx=0x7f89b4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:23.987 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:22.878+0000 7f89beffd700 1 -- 172.21.15.60:0/3793435008 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f89b0020080 con 0x7f89c0106a90 2023-07-20T01:21:23.987 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:23.263+0000 7f89c8061700 1 -- 172.21.15.60:0/3793435008 --> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f89c01018d0 con 0x7f89a8038740 2023-07-20T01:21:23.987 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:23.268+0000 7f89beffd700 1 -- 172.21.15.60:0/3793435008 <== mgr.14120 v2:172.21.15.60:6800/712953724 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f89c01018d0 con 0x7f89a8038740 2023-07-20T01:21:23.988 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:23.270+0000 7f89c8061700 1 -- 172.21.15.60:0/3793435008 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f89a8038740 msgr2=0x7f89a803ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:23.988 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:23.270+0000 7f89c8061700 1 --2- 172.21.15.60:0/3793435008 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f89a8038740 0x7f89a803ac00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f89b4006fd0 tx=0x7f89b4006e40 comp rx=0 tx=0).stop 2023-07-20T01:21:23.988 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:23.270+0000 7f89c8061700 1 -- 172.21.15.60:0/3793435008 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89c0106a90 msgr2=0x7f89c01063f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:23.988 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:23.270+0000 7f89c8061700 1 --2- 172.21.15.60:0/3793435008 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89c0106a90 0x7f89c01063f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f89b00072a0 tx=0x7f89b0003d00 comp rx=0 tx=0).stop 2023-07-20T01:21:23.988 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:23.270+0000 7f89c8061700 1 -- 172.21.15.60:0/3793435008 shutdown_connections 2023-07-20T01:21:23.988 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:23.270+0000 7f89c8061700 1 --2- 172.21.15.60:0/3793435008 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f89a8038740 0x7f89a803ac00 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:23.988 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:23.270+0000 7f89c8061700 1 --2- 172.21.15.60:0/3793435008 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89c0106a90 0x7f89c01063f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:23.989 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:23.270+0000 7f89c8061700 1 -- 172.21.15.60:0/3793435008 >> 172.21.15.60:0/3793435008 conn(0x7f89c0100480 msgr2=0x7f89c01011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:23.989 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:23.270+0000 7f89c8061700 1 -- 172.21.15.60:0/3793435008 shutdown_connections 2023-07-20T01:21:23.989 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:23.270+0000 7f89c8061700 1 -- 172.21.15.60:0/3793435008 wait complete. 2023-07-20T01:21:23.989 INFO:teuthology.orchestra.run.smithi060.stdout:Deploying unmanaged mgr service... 2023-07-20T01:21:24.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:24 smithi060 ceph-mon[96524]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:21:24.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:24 smithi060 ceph-mon[96524]: Saving service mon spec with placement count:5 2023-07-20T01:21:24.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:24 smithi060 ceph-mon[96524]: from='mgr.14120 172.21.15.60:0/4194926595' entity='mgr.x' 2023-07-20T01:21:24.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:24 smithi060 ceph-mon[96524]: from='mgr.14120 172.21.15.60:0/4194926595' entity='mgr.x' 2023-07-20T01:21:25.328 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-07-20T01:21:25.328 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.458+0000 7f86a1916700 1 Processor -- start 2023-07-20T01:21:25.328 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.458+0000 7f86a1916700 1 -- start start 2023-07-20T01:21:25.329 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.458+0000 7f86a1916700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f869c106a80 0x7f869c108e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:25.329 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.458+0000 7f86a1916700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c079dd0 con 0x7f869c106a80 2023-07-20T01:21:25.329 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.458+0000 7f869affd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f869c106a80 0x7f869c108e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:25.329 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.458+0000 7f869affd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f869c106a80 0x7f869c108e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41988/0 (socket says 172.21.15.60:41988) 2023-07-20T01:21:25.329 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.458+0000 7f869affd700 1 -- 172.21.15.60:0/975172819 learned_addr learned my addr 172.21.15.60:0/975172819 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:25.330 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.459+0000 7f869affd700 1 -- 172.21.15.60:0/975172819 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869c1093b0 con 0x7f869c106a80 2023-07-20T01:21:25.330 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.459+0000 7f869affd700 1 --2- 172.21.15.60:0/975172819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f869c106a80 0x7f869c108e70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f868c0098a0 tx=0x7f868c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41de0e732b965067 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:25.330 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.459+0000 7f8699ffb700 1 -- 172.21.15.60:0/975172819 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f868c004160 con 0x7f869c106a80 2023-07-20T01:21:25.330 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.459+0000 7f8699ffb700 1 -- 172.21.15.60:0/975172819 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f868c0042c0 con 0x7f869c106a80 2023-07-20T01:21:25.330 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.459+0000 7f8699ffb700 1 -- 172.21.15.60:0/975172819 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f868c0118e0 con 0x7f869c106a80 2023-07-20T01:21:25.330 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.460+0000 7f86a1916700 1 -- 172.21.15.60:0/975172819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f869c106a80 msgr2=0x7f869c108e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:25.330 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.460+0000 7f86a1916700 1 --2- 172.21.15.60:0/975172819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f869c106a80 0x7f869c108e70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f868c0098a0 tx=0x7f868c009bb0 comp rx=0 tx=0).stop 2023-07-20T01:21:25.331 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.460+0000 7f86a1916700 1 -- 172.21.15.60:0/975172819 shutdown_connections 2023-07-20T01:21:25.331 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.460+0000 7f86a1916700 1 --2- 172.21.15.60:0/975172819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f869c106a80 0x7f869c108e70 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:25.331 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.460+0000 7f86a1916700 1 -- 172.21.15.60:0/975172819 >> 172.21.15.60:0/975172819 conn(0x7f869c100410 msgr2=0x7f869c102870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:25.331 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.460+0000 7f86a1916700 1 -- 172.21.15.60:0/975172819 shutdown_connections 2023-07-20T01:21:25.331 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.460+0000 7f86a1916700 1 -- 172.21.15.60:0/975172819 wait complete. 2023-07-20T01:21:25.331 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.460+0000 7f86a1916700 1 Processor -- start 2023-07-20T01:21:25.331 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.461+0000 7f86a1916700 1 -- start start 2023-07-20T01:21:25.332 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.461+0000 7f86a1916700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f869c106a80 0x7f869c199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:25.332 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.461+0000 7f86a1916700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c19a170 con 0x7f869c106a80 2023-07-20T01:21:25.332 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.461+0000 7f869affd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f869c106a80 0x7f869c199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:25.332 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.461+0000 7f869affd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f869c106a80 0x7f869c199c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41990/0 (socket says 172.21.15.60:41990) 2023-07-20T01:21:25.332 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.461+0000 7f869affd700 1 -- 172.21.15.60:0/1248684225 learned_addr learned my addr 172.21.15.60:0/1248684225 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:25.332 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.461+0000 7f869affd700 1 -- 172.21.15.60:0/1248684225 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f868c009550 con 0x7f869c106a80 2023-07-20T01:21:25.332 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.462+0000 7f869affd700 1 --2- 172.21.15.60:0/1248684225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f869c106a80 0x7f869c199c30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f868c006af0 tx=0x7f868c011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:25.333 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.462+0000 7f86a0914700 1 -- 172.21.15.60:0/1248684225 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f868c0037d0 con 0x7f869c106a80 2023-07-20T01:21:25.333 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.462+0000 7f86a0914700 1 -- 172.21.15.60:0/1248684225 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f868c01b440 con 0x7f869c106a80 2023-07-20T01:21:25.333 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.462+0000 7f86a1916700 1 -- 172.21.15.60:0/1248684225 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f869c19a370 con 0x7f869c106a80 2023-07-20T01:21:25.333 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.462+0000 7f86a0914700 1 -- 172.21.15.60:0/1248684225 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f868c0193d0 con 0x7f869c106a80 2023-07-20T01:21:25.333 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.462+0000 7f86a1916700 1 -- 172.21.15.60:0/1248684225 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f869c19a790 con 0x7f869c106a80 2023-07-20T01:21:25.333 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.463+0000 7f86a0914700 1 -- 172.21.15.60:0/1248684225 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f868c019690 con 0x7f869c106a80 2023-07-20T01:21:25.333 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.463+0000 7f86a1916700 1 -- 172.21.15.60:0/1248684225 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f869c10a0e0 con 0x7f869c106a80 2023-07-20T01:21:25.334 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.463+0000 7f86a0914700 1 --2- 172.21.15.60:0/1248684225 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f86840386c0 0x7f868403ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:25.334 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.463+0000 7f86a0914700 1 -- 172.21.15.60:0/1248684225 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f868c04c580 con 0x7f869c106a80 2023-07-20T01:21:25.334 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.463+0000 7f869a7fc700 1 --2- 172.21.15.60:0/1248684225 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f86840386c0 0x7f868403ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:25.334 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.464+0000 7f869a7fc700 1 --2- 172.21.15.60:0/1248684225 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f86840386c0 0x7f868403ab80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8690006fd0 tx=0x7f8690006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:25.334 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.469+0000 7f86a0914700 1 -- 172.21.15.60:0/1248684225 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f868c018db0 con 0x7f869c106a80 2023-07-20T01:21:25.334 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.852+0000 7f86a1916700 1 -- 172.21.15.60:0/1248684225 --> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f869c19d5b0 con 0x7f86840386c0 2023-07-20T01:21:25.335 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.856+0000 7f86a0914700 1 -- 172.21.15.60:0/1248684225 <== mgr.14120 v2:172.21.15.60:6800/712953724 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f869c19d5b0 con 0x7f86840386c0 2023-07-20T01:21:25.335 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.859+0000 7f86a1916700 1 -- 172.21.15.60:0/1248684225 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f86840386c0 msgr2=0x7f868403ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:25.335 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.859+0000 7f86a1916700 1 --2- 172.21.15.60:0/1248684225 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f86840386c0 0x7f868403ab80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8690006fd0 tx=0x7f8690006e40 comp rx=0 tx=0).stop 2023-07-20T01:21:25.335 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.860+0000 7f86a1916700 1 -- 172.21.15.60:0/1248684225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f869c106a80 msgr2=0x7f869c199c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:25.335 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.860+0000 7f86a1916700 1 --2- 172.21.15.60:0/1248684225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f869c106a80 0x7f869c199c30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f868c006af0 tx=0x7f868c011e80 comp rx=0 tx=0).stop 2023-07-20T01:21:25.335 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.860+0000 7f86a1916700 1 -- 172.21.15.60:0/1248684225 shutdown_connections 2023-07-20T01:21:25.336 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.860+0000 7f86a1916700 1 --2- 172.21.15.60:0/1248684225 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f86840386c0 0x7f868403ab80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:25.336 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.860+0000 7f86a1916700 1 --2- 172.21.15.60:0/1248684225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f869c106a80 0x7f869c199c30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:25.336 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.860+0000 7f86a1916700 1 -- 172.21.15.60:0/1248684225 >> 172.21.15.60:0/1248684225 conn(0x7f869c100410 msgr2=0x7f869c101ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:25.336 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.860+0000 7f86a1916700 1 -- 172.21.15.60:0/1248684225 shutdown_connections 2023-07-20T01:21:25.336 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:24.860+0000 7f86a1916700 1 -- 172.21.15.60:0/1248684225 wait complete. 2023-07-20T01:21:25.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:25 smithi060 ceph-mon[96524]: from='mgr.14120 172.21.15.60:0/4194926595' entity='mgr.x' 2023-07-20T01:21:25.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:25 smithi060 ceph-mon[96524]: from='mgr.14120 172.21.15.60:0/4194926595' entity='mgr.x' 2023-07-20T01:21:26.596 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:26 smithi060 ceph-mon[96524]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:21:26.596 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:26 smithi060 ceph-mon[96524]: Saving service mgr spec with placement count:2 2023-07-20T01:21:26.596 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:26 smithi060 ceph-mon[96524]: from='mgr.14120 172.21.15.60:0/4194926595' entity='mgr.x' 2023-07-20T01:21:26.823 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.041+0000 7f0b83dae700 1 Processor -- start 2023-07-20T01:21:26.823 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.041+0000 7f0b83dae700 1 -- start start 2023-07-20T01:21:26.823 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.041+0000 7f0b83dae700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b7c105fc0 0x7f0b7c1063e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:26.823 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.041+0000 7f0b83dae700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c1069b0 con 0x7f0b7c105fc0 2023-07-20T01:21:26.823 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.042+0000 7f0b81b4a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b7c105fc0 0x7f0b7c1063e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:26.824 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.042+0000 7f0b81b4a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b7c105fc0 0x7f0b7c1063e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41994/0 (socket says 172.21.15.60:41994) 2023-07-20T01:21:26.824 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.042+0000 7f0b81b4a700 1 -- 172.21.15.60:0/2364408451 learned_addr learned my addr 172.21.15.60:0/2364408451 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:26.824 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.042+0000 7f0b81b4a700 1 -- 172.21.15.60:0/2364408451 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7c1071c0 con 0x7f0b7c105fc0 2023-07-20T01:21:26.824 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.042+0000 7f0b81b4a700 1 --2- 172.21.15.60:0/2364408451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b7c105fc0 0x7f0b7c1063e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0b780098a0 tx=0x7f0b78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fb448532b206748 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:26.824 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.042+0000 7f0b80b48700 1 -- 172.21.15.60:0/2364408451 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b78004160 con 0x7f0b7c105fc0 2023-07-20T01:21:26.824 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.042+0000 7f0b80b48700 1 -- 172.21.15.60:0/2364408451 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0b780042c0 con 0x7f0b7c105fc0 2023-07-20T01:21:26.824 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.042+0000 7f0b80b48700 1 -- 172.21.15.60:0/2364408451 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b780118e0 con 0x7f0b7c105fc0 2023-07-20T01:21:26.825 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.043+0000 7f0b83dae700 1 -- 172.21.15.60:0/2364408451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b7c105fc0 msgr2=0x7f0b7c1063e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:26.825 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.043+0000 7f0b83dae700 1 --2- 172.21.15.60:0/2364408451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b7c105fc0 0x7f0b7c1063e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0b780098a0 tx=0x7f0b78009bb0 comp rx=0 tx=0).stop 2023-07-20T01:21:26.825 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.043+0000 7f0b83dae700 1 -- 172.21.15.60:0/2364408451 shutdown_connections 2023-07-20T01:21:26.825 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.043+0000 7f0b83dae700 1 --2- 172.21.15.60:0/2364408451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b7c105fc0 0x7f0b7c1063e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:26.825 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.043+0000 7f0b83dae700 1 -- 172.21.15.60:0/2364408451 >> 172.21.15.60:0/2364408451 conn(0x7f0b7c101580 msgr2=0x7f0b7c1039a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:26.825 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.043+0000 7f0b83dae700 1 -- 172.21.15.60:0/2364408451 shutdown_connections 2023-07-20T01:21:26.825 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.043+0000 7f0b83dae700 1 -- 172.21.15.60:0/2364408451 wait complete. 2023-07-20T01:21:26.825 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.044+0000 7f0b83dae700 1 Processor -- start 2023-07-20T01:21:26.826 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.044+0000 7f0b83dae700 1 -- start start 2023-07-20T01:21:26.826 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.044+0000 7f0b83dae700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b7c105fc0 0x7f0b7c199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:26.826 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.044+0000 7f0b83dae700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c19a2e0 con 0x7f0b7c105fc0 2023-07-20T01:21:26.826 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.044+0000 7f0b81b4a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b7c105fc0 0x7f0b7c199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:26.826 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.044+0000 7f0b81b4a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b7c105fc0 0x7f0b7c199da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41996/0 (socket says 172.21.15.60:41996) 2023-07-20T01:21:26.826 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.044+0000 7f0b81b4a700 1 -- 172.21.15.60:0/2473253625 learned_addr learned my addr 172.21.15.60:0/2473253625 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:26.826 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.045+0000 7f0b81b4a700 1 -- 172.21.15.60:0/2473253625 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b78009550 con 0x7f0b7c105fc0 2023-07-20T01:21:26.827 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.045+0000 7f0b81b4a700 1 --2- 172.21.15.60:0/2473253625 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b7c105fc0 0x7f0b7c199da0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0b78006af0 tx=0x7f0b78011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:26.827 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.045+0000 7f0b72ffd700 1 -- 172.21.15.60:0/2473253625 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b780037d0 con 0x7f0b7c105fc0 2023-07-20T01:21:26.827 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.045+0000 7f0b72ffd700 1 -- 172.21.15.60:0/2473253625 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0b7801b440 con 0x7f0b7c105fc0 2023-07-20T01:21:26.827 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.045+0000 7f0b83dae700 1 -- 172.21.15.60:0/2473253625 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b7c19a4e0 con 0x7f0b7c105fc0 2023-07-20T01:21:26.827 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.045+0000 7f0b72ffd700 1 -- 172.21.15.60:0/2473253625 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b78023400 con 0x7f0b7c105fc0 2023-07-20T01:21:26.827 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.045+0000 7f0b83dae700 1 -- 172.21.15.60:0/2473253625 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b7c19a900 con 0x7f0b7c105fc0 2023-07-20T01:21:26.827 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.046+0000 7f0b72ffd700 1 -- 172.21.15.60:0/2473253625 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f0b78023670 con 0x7f0b7c105fc0 2023-07-20T01:21:26.827 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.046+0000 7f0b72ffd700 1 --2- 172.21.15.60:0/2473253625 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f0b68038710 0x7f0b6803abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:26.828 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.046+0000 7f0b72ffd700 1 -- 172.21.15.60:0/2473253625 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f0b7801b9f0 con 0x7f0b7c105fc0 2023-07-20T01:21:26.828 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.046+0000 7f0b81349700 1 --2- 172.21.15.60:0/2473253625 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f0b68038710 0x7f0b6803abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:26.828 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.046+0000 7f0b83dae700 1 -- 172.21.15.60:0/2473253625 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b7c193da0 con 0x7f0b7c105fc0 2023-07-20T01:21:26.828 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.047+0000 7f0b81349700 1 --2- 172.21.15.60:0/2473253625 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f0b68038710 0x7f0b6803abd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c006fd0 tx=0x7f0b6c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:26.828 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.051+0000 7f0b72ffd700 1 -- 172.21.15.60:0/2473253625 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0b78023920 con 0x7f0b7c105fc0 2023-07-20T01:21:26.828 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.421+0000 7f0b83dae700 1 -- 172.21.15.60:0/2473253625 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f0b7c0008d0 con 0x7f0b7c105fc0 2023-07-20T01:21:26.828 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.424+0000 7f0b72ffd700 1 -- 172.21.15.60:0/2473253625 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f0b7802bb40 con 0x7f0b7c105fc0 2023-07-20T01:21:26.829 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.427+0000 7f0b83dae700 1 -- 172.21.15.60:0/2473253625 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f0b68038710 msgr2=0x7f0b6803abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:26.829 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.427+0000 7f0b83dae700 1 --2- 172.21.15.60:0/2473253625 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f0b68038710 0x7f0b6803abd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c006fd0 tx=0x7f0b6c006e40 comp rx=0 tx=0).stop 2023-07-20T01:21:26.829 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.427+0000 7f0b83dae700 1 -- 172.21.15.60:0/2473253625 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b7c105fc0 msgr2=0x7f0b7c199da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:26.829 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.427+0000 7f0b83dae700 1 --2- 172.21.15.60:0/2473253625 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b7c105fc0 0x7f0b7c199da0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0b78006af0 tx=0x7f0b78011ed0 comp rx=0 tx=0).stop 2023-07-20T01:21:26.829 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.427+0000 7f0b83dae700 1 -- 172.21.15.60:0/2473253625 shutdown_connections 2023-07-20T01:21:26.829 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.427+0000 7f0b83dae700 1 --2- 172.21.15.60:0/2473253625 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f0b68038710 0x7f0b6803abd0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:26.829 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.427+0000 7f0b83dae700 1 --2- 172.21.15.60:0/2473253625 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b7c105fc0 0x7f0b7c199da0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:26.829 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.427+0000 7f0b83dae700 1 -- 172.21.15.60:0/2473253625 >> 172.21.15.60:0/2473253625 conn(0x7f0b7c101580 msgr2=0x7f0b7c103950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:26.830 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.427+0000 7f0b83dae700 1 -- 172.21.15.60:0/2473253625 shutdown_connections 2023-07-20T01:21:26.830 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:26.427+0000 7f0b83dae700 1 -- 172.21.15.60:0/2473253625 wait complete. 2023-07-20T01:21:27.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:27 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/2473253625' entity='client.admin' 2023-07-20T01:21:27.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:27 smithi060 ceph-mon[96524]: from='mgr.14120 172.21.15.60:0/4194926595' entity='mgr.x' 2023-07-20T01:21:28.570 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.338+0000 7f5c44929700 1 Processor -- start 2023-07-20T01:21:28.570 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.338+0000 7f5c44929700 1 -- start start 2023-07-20T01:21:28.570 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.339+0000 7f5c44929700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c40104880 0x7f5c40104ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:28.570 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.339+0000 7f5c44929700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c40079dd0 con 0x7f5c40104880 2023-07-20T01:21:28.571 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.339+0000 7f5c3e59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c40104880 0x7f5c40104ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:28.571 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.339+0000 7f5c3e59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c40104880 0x7f5c40104ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42000/0 (socket says 172.21.15.60:42000) 2023-07-20T01:21:28.571 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.339+0000 7f5c3e59c700 1 -- 172.21.15.60:0/1661532972 learned_addr learned my addr 172.21.15.60:0/1661532972 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:28.571 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.339+0000 7f5c3e59c700 1 -- 172.21.15.60:0/1661532972 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c401051e0 con 0x7f5c40104880 2023-07-20T01:21:28.571 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.339+0000 7f5c3e59c700 1 --2- 172.21.15.60:0/1661532972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c40104880 0x7f5c40104ca0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5c340098a0 tx=0x7f5c34009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8043491181d962ab server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:28.571 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.339+0000 7f5c3d59a700 1 -- 172.21.15.60:0/1661532972 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5c34004160 con 0x7f5c40104880 2023-07-20T01:21:28.571 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.339+0000 7f5c3d59a700 1 -- 172.21.15.60:0/1661532972 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5c340042c0 con 0x7f5c40104880 2023-07-20T01:21:28.572 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.339+0000 7f5c3d59a700 1 -- 172.21.15.60:0/1661532972 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5c340118e0 con 0x7f5c40104880 2023-07-20T01:21:28.572 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.340+0000 7f5c44929700 1 -- 172.21.15.60:0/1661532972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c40104880 msgr2=0x7f5c40104ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:28.572 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.340+0000 7f5c44929700 1 --2- 172.21.15.60:0/1661532972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c40104880 0x7f5c40104ca0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5c340098a0 tx=0x7f5c34009bb0 comp rx=0 tx=0).stop 2023-07-20T01:21:28.572 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.340+0000 7f5c44929700 1 -- 172.21.15.60:0/1661532972 shutdown_connections 2023-07-20T01:21:28.572 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.340+0000 7f5c44929700 1 --2- 172.21.15.60:0/1661532972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c40104880 0x7f5c40104ca0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:28.572 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.340+0000 7f5c44929700 1 -- 172.21.15.60:0/1661532972 >> 172.21.15.60:0/1661532972 conn(0x7f5c40100480 msgr2=0x7f5c401028a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:28.572 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.340+0000 7f5c44929700 1 -- 172.21.15.60:0/1661532972 shutdown_connections 2023-07-20T01:21:28.573 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.340+0000 7f5c44929700 1 -- 172.21.15.60:0/1661532972 wait complete. 2023-07-20T01:21:28.573 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.341+0000 7f5c44929700 1 Processor -- start 2023-07-20T01:21:28.573 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.341+0000 7f5c44929700 1 -- start start 2023-07-20T01:21:28.573 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.341+0000 7f5c44929700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c40104880 0x7f5c4019bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:28.573 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.341+0000 7f5c44929700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4019c310 con 0x7f5c40104880 2023-07-20T01:21:28.573 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.341+0000 7f5c3e59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c40104880 0x7f5c4019bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:28.573 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.341+0000 7f5c3e59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c40104880 0x7f5c4019bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42002/0 (socket says 172.21.15.60:42002) 2023-07-20T01:21:28.574 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.341+0000 7f5c3e59c700 1 -- 172.21.15.60:0/2340065477 learned_addr learned my addr 172.21.15.60:0/2340065477 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:28.574 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.342+0000 7f5c3e59c700 1 -- 172.21.15.60:0/2340065477 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c34009550 con 0x7f5c40104880 2023-07-20T01:21:28.574 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.342+0000 7f5c3e59c700 1 --2- 172.21.15.60:0/2340065477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c40104880 0x7f5c4019bdd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5c340118f0 tx=0x7f5c340119d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:28.574 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.342+0000 7f5c2f7fe700 1 -- 172.21.15.60:0/2340065477 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5c340036a0 con 0x7f5c40104880 2023-07-20T01:21:28.574 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.342+0000 7f5c2f7fe700 1 -- 172.21.15.60:0/2340065477 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5c3401b440 con 0x7f5c40104880 2023-07-20T01:21:28.574 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.342+0000 7f5c44929700 1 -- 172.21.15.60:0/2340065477 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c4019c510 con 0x7f5c40104880 2023-07-20T01:21:28.574 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.342+0000 7f5c2f7fe700 1 -- 172.21.15.60:0/2340065477 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5c340193f0 con 0x7f5c40104880 2023-07-20T01:21:28.574 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.342+0000 7f5c44929700 1 -- 172.21.15.60:0/2340065477 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c4019c930 con 0x7f5c40104880 2023-07-20T01:21:28.575 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.343+0000 7f5c2f7fe700 1 -- 172.21.15.60:0/2340065477 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f5c34003800 con 0x7f5c40104880 2023-07-20T01:21:28.575 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.343+0000 7f5c44929700 1 -- 172.21.15.60:0/2340065477 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c40195680 con 0x7f5c40104880 2023-07-20T01:21:28.575 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.343+0000 7f5c2f7fe700 1 --2- 172.21.15.60:0/2340065477 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f5c28038710 0x7f5c2803abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:28.575 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.343+0000 7f5c2f7fe700 1 -- 172.21.15.60:0/2340065477 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f5c3404c640 con 0x7f5c40104880 2023-07-20T01:21:28.575 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.343+0000 7f5c3dd9b700 1 --2- 172.21.15.60:0/2340065477 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f5c28038710 0x7f5c2803abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:28.575 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.344+0000 7f5c3dd9b700 1 --2- 172.21.15.60:0/2340065477 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f5c28038710 0x7f5c2803abd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5c30006fd0 tx=0x7f5c30006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:28.575 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.348+0000 7f5c2f7fe700 1 -- 172.21.15.60:0/2340065477 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5c34023590 con 0x7f5c40104880 2023-07-20T01:21:28.576 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.718+0000 7f5c44929700 1 -- 172.21.15.60:0/2340065477 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f5c4002c970 con 0x7f5c40104880 2023-07-20T01:21:28.576 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.721+0000 7f5c2f7fe700 1 -- 172.21.15.60:0/2340065477 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f5c3401f070 con 0x7f5c40104880 2023-07-20T01:21:28.576 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.723+0000 7f5c44929700 1 -- 172.21.15.60:0/2340065477 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f5c28038710 msgr2=0x7f5c2803abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:28.576 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.723+0000 7f5c44929700 1 --2- 172.21.15.60:0/2340065477 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f5c28038710 0x7f5c2803abd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5c30006fd0 tx=0x7f5c30006e40 comp rx=0 tx=0).stop 2023-07-20T01:21:28.576 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.723+0000 7f5c44929700 1 -- 172.21.15.60:0/2340065477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c40104880 msgr2=0x7f5c4019bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:28.576 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.723+0000 7f5c44929700 1 --2- 172.21.15.60:0/2340065477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c40104880 0x7f5c4019bdd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5c340118f0 tx=0x7f5c340119d0 comp rx=0 tx=0).stop 2023-07-20T01:21:28.576 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.723+0000 7f5c44929700 1 -- 172.21.15.60:0/2340065477 shutdown_connections 2023-07-20T01:21:28.577 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.723+0000 7f5c44929700 1 --2- 172.21.15.60:0/2340065477 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f5c28038710 0x7f5c2803abd0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:28.577 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.723+0000 7f5c44929700 1 --2- 172.21.15.60:0/2340065477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c40104880 0x7f5c4019bdd0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:28.577 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.723+0000 7f5c44929700 1 -- 172.21.15.60:0/2340065477 >> 172.21.15.60:0/2340065477 conn(0x7f5c40100480 msgr2=0x7f5c40190440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:28.577 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.723+0000 7f5c44929700 1 -- 172.21.15.60:0/2340065477 shutdown_connections 2023-07-20T01:21:28.577 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:27.723+0000 7f5c44929700 1 -- 172.21.15.60:0/2340065477 wait complete. 2023-07-20T01:21:28.577 INFO:teuthology.orchestra.run.smithi060.stdout:Enabling the dashboard module... 2023-07-20T01:21:28.801 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:28 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/2340065477' entity='client.admin' 2023-07-20T01:21:30.016 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:29 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/46381697' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-07-20T01:21:30.017 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:29 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: ignoring --setuser ceph since I am not root 2023-07-20T01:21:30.017 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:29 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: ignoring --setgroup ceph since I am not root 2023-07-20T01:21:30.017 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:29 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:29.937+0000 7f2d6f621200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T01:21:30.017 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:30 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:30.034+0000 7f2d6f621200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T01:21:30.114 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.091+0000 7fc6c6cf2700 1 Processor -- start 2023-07-20T01:21:30.118 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.091+0000 7fc6c6cf2700 1 -- start start 2023-07-20T01:21:30.118 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.092+0000 7fc6c6cf2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6c0106b10 0x7fc6c0108f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:30.118 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.092+0000 7fc6c6cf2700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c0073e20 con 0x7fc6c0106b10 2023-07-20T01:21:30.118 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.092+0000 7fc6c4a8e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6c0106b10 0x7fc6c0108f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:30.119 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.092+0000 7fc6c4a8e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6c0106b10 0x7fc6c0108f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42006/0 (socket says 172.21.15.60:42006) 2023-07-20T01:21:30.119 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.092+0000 7fc6c4a8e700 1 -- 172.21.15.60:0/3205061365 learned_addr learned my addr 172.21.15.60:0/3205061365 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:30.119 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.092+0000 7fc6c4a8e700 1 -- 172.21.15.60:0/3205061365 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c0073f60 con 0x7fc6c0106b10 2023-07-20T01:21:30.119 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.092+0000 7fc6c4a8e700 1 --2- 172.21.15.60:0/3205061365 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6c0106b10 0x7fc6c0108f00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc6b40098a0 tx=0x7fc6b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c13be058ff1fbb6d server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:30.119 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.093+0000 7fc6bf7fe700 1 -- 172.21.15.60:0/3205061365 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc6b4004160 con 0x7fc6c0106b10 2023-07-20T01:21:30.119 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.093+0000 7fc6bf7fe700 1 -- 172.21.15.60:0/3205061365 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc6b40042c0 con 0x7fc6c0106b10 2023-07-20T01:21:30.119 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.093+0000 7fc6bf7fe700 1 -- 172.21.15.60:0/3205061365 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc6b40118e0 con 0x7fc6c0106b10 2023-07-20T01:21:30.120 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.093+0000 7fc6c6cf2700 1 -- 172.21.15.60:0/3205061365 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6c0106b10 msgr2=0x7fc6c0108f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:30.120 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.093+0000 7fc6c6cf2700 1 --2- 172.21.15.60:0/3205061365 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6c0106b10 0x7fc6c0108f00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc6b40098a0 tx=0x7fc6b4009bb0 comp rx=0 tx=0).stop 2023-07-20T01:21:30.120 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.093+0000 7fc6c6cf2700 1 -- 172.21.15.60:0/3205061365 shutdown_connections 2023-07-20T01:21:30.120 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.093+0000 7fc6c6cf2700 1 --2- 172.21.15.60:0/3205061365 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6c0106b10 0x7fc6c0108f00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:30.120 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.093+0000 7fc6c6cf2700 1 -- 172.21.15.60:0/3205061365 >> 172.21.15.60:0/3205061365 conn(0x7fc6c01004a0 msgr2=0x7fc6c01028c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:30.120 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.093+0000 7fc6c6cf2700 1 -- 172.21.15.60:0/3205061365 shutdown_connections 2023-07-20T01:21:30.120 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.093+0000 7fc6c6cf2700 1 -- 172.21.15.60:0/3205061365 wait complete. 2023-07-20T01:21:30.121 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.094+0000 7fc6c6cf2700 1 Processor -- start 2023-07-20T01:21:30.121 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.094+0000 7fc6c6cf2700 1 -- start start 2023-07-20T01:21:30.121 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.094+0000 7fc6c6cf2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6c0106b10 0x7fc6c0199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:30.121 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.094+0000 7fc6c6cf2700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c019a1c0 con 0x7fc6c0106b10 2023-07-20T01:21:30.121 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.094+0000 7fc6c4a8e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6c0106b10 0x7fc6c0199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:30.121 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.094+0000 7fc6c4a8e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6c0106b10 0x7fc6c0199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42008/0 (socket says 172.21.15.60:42008) 2023-07-20T01:21:30.121 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.094+0000 7fc6c4a8e700 1 -- 172.21.15.60:0/46381697 learned_addr learned my addr 172.21.15.60:0/46381697 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:30.121 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.095+0000 7fc6c4a8e700 1 -- 172.21.15.60:0/46381697 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b4009550 con 0x7fc6c0106b10 2023-07-20T01:21:30.122 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.095+0000 7fc6c4a8e700 1 --2- 172.21.15.60:0/46381697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6c0106b10 0x7fc6c0199c80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc6b4003e30 tx=0x7fc6b40119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:30.122 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.095+0000 7fc6bdffb700 1 -- 172.21.15.60:0/46381697 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc6b40036a0 con 0x7fc6c0106b10 2023-07-20T01:21:30.122 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.095+0000 7fc6bdffb700 1 -- 172.21.15.60:0/46381697 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc6b401b440 con 0x7fc6c0106b10 2023-07-20T01:21:30.122 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.095+0000 7fc6c6cf2700 1 -- 172.21.15.60:0/46381697 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6c019a3c0 con 0x7fc6c0106b10 2023-07-20T01:21:30.122 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.095+0000 7fc6bdffb700 1 -- 172.21.15.60:0/46381697 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc6b4023400 con 0x7fc6c0106b10 2023-07-20T01:21:30.122 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.095+0000 7fc6c6cf2700 1 -- 172.21.15.60:0/46381697 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6c019a7e0 con 0x7fc6c0106b10 2023-07-20T01:21:30.122 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.096+0000 7fc6bdffb700 1 -- 172.21.15.60:0/46381697 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fc6b4023680 con 0x7fc6c0106b10 2023-07-20T01:21:30.122 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.096+0000 7fc6bdffb700 1 --2- 172.21.15.60:0/46381697 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7fc6b0038710 0x7fc6b003abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:30.123 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.096+0000 7fc6bdffb700 1 -- 172.21.15.60:0/46381697 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc6b404c870 con 0x7fc6c0106b10 2023-07-20T01:21:30.123 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.096+0000 7fc6c6cf2700 1 -- 172.21.15.60:0/46381697 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6c0061da0 con 0x7fc6c0106b10 2023-07-20T01:21:30.123 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.096+0000 7fc6bffff700 1 --2- 172.21.15.60:0/46381697 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7fc6b0038710 0x7fc6b003abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:30.123 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.097+0000 7fc6bffff700 1 --2- 172.21.15.60:0/46381697 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7fc6b0038710 0x7fc6b003abd0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac006fd0 tx=0x7fc6ac006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:30.123 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.101+0000 7fc6bdffb700 1 -- 172.21.15.60:0/46381697 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc6b402c3f0 con 0x7fc6c0106b10 2023-07-20T01:21:30.123 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.569+0000 7fc6c6cf2700 1 -- 172.21.15.60:0/46381697 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fc6c019d300 con 0x7fc6c0106b10 2023-07-20T01:21:30.123 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.723+0000 7fc6bdffb700 1 -- 172.21.15.60:0/46381697 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7fc6b40038a0 con 0x7fc6c0106b10 2023-07-20T01:21:30.124 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.725+0000 7fc6bdffb700 1 -- 172.21.15.60:0/46381697 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fc6b402b350 con 0x7fc6c0106b10 2023-07-20T01:21:30.124 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.728+0000 7fc6c6cf2700 1 -- 172.21.15.60:0/46381697 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7fc6b0038710 msgr2=0x7fc6b003abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:30.124 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.728+0000 7fc6c6cf2700 1 --2- 172.21.15.60:0/46381697 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7fc6b0038710 0x7fc6b003abd0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac006fd0 tx=0x7fc6ac006e40 comp rx=0 tx=0).stop 2023-07-20T01:21:30.124 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.728+0000 7fc6c6cf2700 1 -- 172.21.15.60:0/46381697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6c0106b10 msgr2=0x7fc6c0199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:30.124 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.728+0000 7fc6c6cf2700 1 --2- 172.21.15.60:0/46381697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6c0106b10 0x7fc6c0199c80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc6b4003e30 tx=0x7fc6b40119a0 comp rx=0 tx=0).stop 2023-07-20T01:21:30.124 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.728+0000 7fc6c6cf2700 1 -- 172.21.15.60:0/46381697 shutdown_connections 2023-07-20T01:21:30.124 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.728+0000 7fc6c6cf2700 1 --2- 172.21.15.60:0/46381697 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7fc6b0038710 0x7fc6b003abd0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:30.125 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.728+0000 7fc6c6cf2700 1 --2- 172.21.15.60:0/46381697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6c0106b10 0x7fc6c0199c80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:30.125 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.728+0000 7fc6c6cf2700 1 -- 172.21.15.60:0/46381697 >> 172.21.15.60:0/46381697 conn(0x7fc6c01004a0 msgr2=0x7fc6c0101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:30.125 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.728+0000 7fc6c6cf2700 1 -- 172.21.15.60:0/46381697 shutdown_connections 2023-07-20T01:21:30.125 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:29.728+0000 7fc6c6cf2700 1 -- 172.21.15.60:0/46381697 wait complete. 2023-07-20T01:21:30.335 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:30 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:30.146+0000 7f2d6f621200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T01:21:30.336 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:30 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:30.267+0000 7f2d6f621200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T01:21:30.679 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:30 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:30.484+0000 7f2d6f621200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T01:21:30.978 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:30 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:30.886+0000 7f2d6f621200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T01:21:30.978 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:30 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/46381697' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-07-20T01:21:30.978 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:30 smithi060 ceph-mon[96524]: mgrmap e9: x(active, since 18s) 2023-07-20T01:21:31.259 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:30 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:30.996+0000 7f2d6f621200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T01:21:31.259 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:31 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:31.149+0000 7f2d6f621200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T01:21:31.568 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout { 2023-07-20T01:21:31.568 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-07-20T01:21:31.569 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-20T01:21:31.569 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-07-20T01:21:31.569 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-20T01:21:31.569 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout } 2023-07-20T01:21:31.569 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.668+0000 7f55fa3f7700 1 Processor -- start 2023-07-20T01:21:31.569 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.668+0000 7f55fa3f7700 1 -- start start 2023-07-20T01:21:31.569 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.668+0000 7f55fa3f7700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55f41048b0 0x7f55f4104cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:31.570 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.668+0000 7f55fa3f7700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f4073e60 con 0x7f55f41048b0 2023-07-20T01:21:31.570 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.668+0000 7f55f3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55f41048b0 0x7f55f4104cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:31.570 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.668+0000 7f55f3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55f41048b0 0x7f55f4104cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42022/0 (socket says 172.21.15.60:42022) 2023-07-20T01:21:31.570 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.668+0000 7f55f3fff700 1 -- 172.21.15.60:0/2042105108 learned_addr learned my addr 172.21.15.60:0/2042105108 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:31.570 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.669+0000 7f55f3fff700 1 -- 172.21.15.60:0/2042105108 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f4073fa0 con 0x7f55f41048b0 2023-07-20T01:21:31.570 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.669+0000 7f55f3fff700 1 --2- 172.21.15.60:0/2042105108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55f41048b0 0x7f55f4104cd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f55e40098a0 tx=0x7f55e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc9fa4f77793c8d6 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:31.570 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.669+0000 7f55f2ffd700 1 -- 172.21.15.60:0/2042105108 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f55e4004160 con 0x7f55f41048b0 2023-07-20T01:21:31.571 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.669+0000 7f55f2ffd700 1 -- 172.21.15.60:0/2042105108 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f55e40042c0 con 0x7f55f41048b0 2023-07-20T01:21:31.571 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.669+0000 7f55f2ffd700 1 -- 172.21.15.60:0/2042105108 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f55e40118e0 con 0x7f55f41048b0 2023-07-20T01:21:31.571 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.670+0000 7f55fa3f7700 1 -- 172.21.15.60:0/2042105108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55f41048b0 msgr2=0x7f55f4104cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:31.571 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.670+0000 7f55fa3f7700 1 --2- 172.21.15.60:0/2042105108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55f41048b0 0x7f55f4104cd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f55e40098a0 tx=0x7f55e4009bb0 comp rx=0 tx=0).stop 2023-07-20T01:21:31.571 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.670+0000 7f55fa3f7700 1 -- 172.21.15.60:0/2042105108 shutdown_connections 2023-07-20T01:21:31.571 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.670+0000 7f55fa3f7700 1 --2- 172.21.15.60:0/2042105108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55f41048b0 0x7f55f4104cd0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:31.571 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.670+0000 7f55fa3f7700 1 -- 172.21.15.60:0/2042105108 >> 172.21.15.60:0/2042105108 conn(0x7f55f4100490 msgr2=0x7f55f41028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:31.572 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.670+0000 7f55fa3f7700 1 -- 172.21.15.60:0/2042105108 shutdown_connections 2023-07-20T01:21:31.572 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.670+0000 7f55fa3f7700 1 -- 172.21.15.60:0/2042105108 wait complete. 2023-07-20T01:21:31.572 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.671+0000 7f55fa3f7700 1 Processor -- start 2023-07-20T01:21:31.572 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.671+0000 7f55fa3f7700 1 -- start start 2023-07-20T01:21:31.572 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.671+0000 7f55fa3f7700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55f41048b0 0x7f55f419be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:31.572 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.671+0000 7f55fa3f7700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f419c3c0 con 0x7f55f41048b0 2023-07-20T01:21:31.572 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.671+0000 7f55f3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55f41048b0 0x7f55f419be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:31.572 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.671+0000 7f55f3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55f41048b0 0x7f55f419be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42024/0 (socket says 172.21.15.60:42024) 2023-07-20T01:21:31.573 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.671+0000 7f55f3fff700 1 -- 172.21.15.60:0/4154023428 learned_addr learned my addr 172.21.15.60:0/4154023428 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:31.573 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.672+0000 7f55f3fff700 1 -- 172.21.15.60:0/4154023428 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e4009550 con 0x7f55f41048b0 2023-07-20T01:21:31.573 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.672+0000 7f55f3fff700 1 --2- 172.21.15.60:0/4154023428 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55f41048b0 0x7f55f419be80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f55e40118f0 tx=0x7f55e40119d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:31.573 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.672+0000 7f55f17fa700 1 -- 172.21.15.60:0/4154023428 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f55e40036a0 con 0x7f55f41048b0 2023-07-20T01:21:31.573 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.672+0000 7f55f17fa700 1 -- 172.21.15.60:0/4154023428 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f55e401b440 con 0x7f55f41048b0 2023-07-20T01:21:31.573 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.672+0000 7f55fa3f7700 1 -- 172.21.15.60:0/4154023428 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55f419c5c0 con 0x7f55f41048b0 2023-07-20T01:21:31.573 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.672+0000 7f55f17fa700 1 -- 172.21.15.60:0/4154023428 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f55e40193f0 con 0x7f55f41048b0 2023-07-20T01:21:31.574 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.672+0000 7f55fa3f7700 1 -- 172.21.15.60:0/4154023428 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55f419c9e0 con 0x7f55f41048b0 2023-07-20T01:21:31.574 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.673+0000 7f55f17fa700 1 -- 172.21.15.60:0/4154023428 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f55e401b5b0 con 0x7f55f41048b0 2023-07-20T01:21:31.574 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.673+0000 7f55fa3f7700 1 -- 172.21.15.60:0/4154023428 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55f4195760 con 0x7f55f41048b0 2023-07-20T01:21:31.574 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.673+0000 7f55f17fa700 1 --2- 172.21.15.60:0/4154023428 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f55dc0387b0 0x7f55dc03ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:31.574 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.673+0000 7f55f17fa700 1 -- 172.21.15.60:0/4154023428 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f55e404c5e0 con 0x7f55f41048b0 2023-07-20T01:21:31.574 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.673+0000 7f55f37fe700 1 -- 172.21.15.60:0/4154023428 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f55dc0387b0 msgr2=0x7f55dc03ac70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/712953724 2023-07-20T01:21:31.574 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.673+0000 7f55f37fe700 1 --2- 172.21.15.60:0/4154023428 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f55dc0387b0 0x7f55dc03ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T01:21:31.574 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.678+0000 7f55f17fa700 1 -- 172.21.15.60:0/4154023428 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f55e4023a00 con 0x7f55f41048b0 2023-07-20T01:21:31.575 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.874+0000 7f55f37fe700 1 -- 172.21.15.60:0/4154023428 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f55dc0387b0 msgr2=0x7f55dc03ac70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/712953724 2023-07-20T01:21:31.575 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:30.874+0000 7f55f37fe700 1 --2- 172.21.15.60:0/4154023428 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f55dc0387b0 0x7f55dc03ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-20T01:21:31.575 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:31.145+0000 7f55fa3f7700 1 -- 172.21.15.60:0/4154023428 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f55f4061da0 con 0x7f55f41048b0 2023-07-20T01:21:31.575 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:31.146+0000 7f55f17fa700 1 -- 172.21.15.60:0/4154023428 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f55e401f020 con 0x7f55f41048b0 2023-07-20T01:21:31.575 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:31.148+0000 7f55fa3f7700 1 -- 172.21.15.60:0/4154023428 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f55dc0387b0 msgr2=0x7f55dc03ac70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T01:21:31.575 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:31.148+0000 7f55fa3f7700 1 --2- 172.21.15.60:0/4154023428 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f55dc0387b0 0x7f55dc03ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:31.575 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:31.148+0000 7f55fa3f7700 1 -- 172.21.15.60:0/4154023428 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55f41048b0 msgr2=0x7f55f419be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:31.575 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:31.148+0000 7f55fa3f7700 1 --2- 172.21.15.60:0/4154023428 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55f41048b0 0x7f55f419be80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f55e40118f0 tx=0x7f55e40119d0 comp rx=0 tx=0).stop 2023-07-20T01:21:31.576 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:31.148+0000 7f55fa3f7700 1 -- 172.21.15.60:0/4154023428 shutdown_connections 2023-07-20T01:21:31.576 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:31.148+0000 7f55fa3f7700 1 --2- 172.21.15.60:0/4154023428 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f55dc0387b0 0x7f55dc03ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:31.576 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:31.148+0000 7f55fa3f7700 1 --2- 172.21.15.60:0/4154023428 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55f41048b0 0x7f55f419be80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:31.576 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:31.148+0000 7f55fa3f7700 1 -- 172.21.15.60:0/4154023428 >> 172.21.15.60:0/4154023428 conn(0x7f55f4100490 msgr2=0x7f55f4190470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:31.576 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:31.148+0000 7f55fa3f7700 1 -- 172.21.15.60:0/4154023428 shutdown_connections 2023-07-20T01:21:31.576 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:31.148+0000 7f55fa3f7700 1 -- 172.21.15.60:0/4154023428 wait complete. 2023-07-20T01:21:31.576 INFO:teuthology.orchestra.run.smithi060.stdout:Waiting for the mgr to restart... 2023-07-20T01:21:31.577 INFO:teuthology.orchestra.run.smithi060.stdout:Waiting for mgr epoch 9... 2023-07-20T01:21:31.679 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:31 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:31.277+0000 7f2d6f621200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T01:21:32.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:31 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/4154023428' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-20T01:21:32.180 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:31 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:31.802+0000 7f2d6f621200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T01:21:32.929 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:32 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:32.495+0000 7f2d6f621200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T01:21:32.930 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:32 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:32.687+0000 7f2d6f621200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T01:21:33.929 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:33 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:33.654+0000 7f2d6f621200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T01:21:33.929 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:33 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:33.765+0000 7f2d6f621200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T01:21:34.320 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:33 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:33.985+0000 7f2d6f621200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T01:21:34.576 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:34 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:34.338+0000 7f2d6f621200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T01:21:34.929 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:34 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:34.594+0000 7f2d6f621200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T01:21:34.929 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:34 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:34.776+0000 7f2d6f621200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T01:21:35.429 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:35 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:35.106+0000 7f2d6f621200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T01:21:36.179 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:35 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:35.933+0000 7f2d6f621200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T01:21:36.179 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:36 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:36.091+0000 7f2d6f621200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T01:21:36.679 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:36 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:36.336+0000 7f2d6f621200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T01:21:36.679 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:36 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:36.473+0000 7f2d6f621200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T01:21:37.179 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:36 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:36.747+0000 7f2d6f621200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T01:21:38.179 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:21:37 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:21:37.789+0000 7f2d6f621200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T01:21:38.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:37 smithi060 ceph-mon[96524]: Active manager daemon x restarted 2023-07-20T01:21:38.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:37 smithi060 ceph-mon[96524]: Activating manager daemon x 2023-07-20T01:21:38.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:37 smithi060 ceph-mon[96524]: osdmap e3: 0 total, 0 up, 0 in 2023-07-20T01:21:38.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:37 smithi060 ceph-mon[96524]: mgrmap e10: x(active, starting, since 0.00302245s) 2023-07-20T01:21:38.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:37 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:21:38.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:37 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-20T01:21:38.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:37 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-20T01:21:38.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:37 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-20T01:21:38.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:37 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-20T01:21:38.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:37 smithi060 ceph-mon[96524]: Manager daemon x is now available 2023-07-20T01:21:38.181 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:37 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:21:38.181 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:37 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:21:39.085 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:38 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-20T01:21:39.086 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:38 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-20T01:21:39.086 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:38 smithi060 ceph-mon[96524]: [20/Jul/2023:01:21:38] ENGINE Bus STARTING 2023-07-20T01:21:39.086 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:38 smithi060 ceph-mon[96524]: [20/Jul/2023:01:21:38] ENGINE Serving on https://172.21.15.60:7150 2023-07-20T01:21:39.087 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:38 smithi060 ceph-mon[96524]: [20/Jul/2023:01:21:38] ENGINE Serving on http://172.21.15.60:8765 2023-07-20T01:21:39.087 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:38 smithi060 ceph-mon[96524]: [20/Jul/2023:01:21:38] ENGINE Bus STARTED 2023-07-20T01:21:39.087 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:38 smithi060 ceph-mon[96524]: mgrmap e11: x(active, since 1.00597s) 2023-07-20T01:21:39.330 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout { 2023-07-20T01:21:39.330 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-07-20T01:21:39.330 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-20T01:21:39.330 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout } 2023-07-20T01:21:39.331 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.047+0000 7f4e7caa0700 1 Processor -- start 2023-07-20T01:21:39.331 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.047+0000 7f4e7caa0700 1 -- start start 2023-07-20T01:21:39.331 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.048+0000 7f4e7caa0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e78106a20 0x7f4e78108e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:39.331 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.048+0000 7f4e7caa0700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e78073f00 con 0x7f4e78106a20 2023-07-20T01:21:39.331 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.048+0000 7f4e7659c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e78106a20 0x7f4e78108e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:39.331 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.048+0000 7f4e7659c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e78106a20 0x7f4e78108e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42032/0 (socket says 172.21.15.60:42032) 2023-07-20T01:21:39.331 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.048+0000 7f4e7659c700 1 -- 172.21.15.60:0/2319241286 learned_addr learned my addr 172.21.15.60:0/2319241286 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:39.332 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.048+0000 7f4e7659c700 1 -- 172.21.15.60:0/2319241286 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e78109350 con 0x7f4e78106a20 2023-07-20T01:21:39.332 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.048+0000 7f4e7659c700 1 --2- 172.21.15.60:0/2319241286 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e78106a20 0x7f4e78108e10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4e680098a0 tx=0x7f4e68009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dce738382cb77cd server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:39.332 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.048+0000 7f4e7559a700 1 -- 172.21.15.60:0/2319241286 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4e68004160 con 0x7f4e78106a20 2023-07-20T01:21:39.332 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.048+0000 7f4e7559a700 1 -- 172.21.15.60:0/2319241286 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4e680042c0 con 0x7f4e78106a20 2023-07-20T01:21:39.332 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.048+0000 7f4e7559a700 1 -- 172.21.15.60:0/2319241286 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4e680118e0 con 0x7f4e78106a20 2023-07-20T01:21:39.332 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.049+0000 7f4e7caa0700 1 -- 172.21.15.60:0/2319241286 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e78106a20 msgr2=0x7f4e78108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:39.332 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.049+0000 7f4e7caa0700 1 --2- 172.21.15.60:0/2319241286 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e78106a20 0x7f4e78108e10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4e680098a0 tx=0x7f4e68009bb0 comp rx=0 tx=0).stop 2023-07-20T01:21:39.333 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.049+0000 7f4e7caa0700 1 -- 172.21.15.60:0/2319241286 shutdown_connections 2023-07-20T01:21:39.333 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.049+0000 7f4e7caa0700 1 --2- 172.21.15.60:0/2319241286 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e78106a20 0x7f4e78108e10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:39.333 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.049+0000 7f4e7caa0700 1 -- 172.21.15.60:0/2319241286 >> 172.21.15.60:0/2319241286 conn(0x7f4e78100490 msgr2=0x7f4e781028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:39.333 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.049+0000 7f4e7caa0700 1 -- 172.21.15.60:0/2319241286 shutdown_connections 2023-07-20T01:21:39.333 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.049+0000 7f4e7caa0700 1 -- 172.21.15.60:0/2319241286 wait complete. 2023-07-20T01:21:39.333 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.050+0000 7f4e7caa0700 1 Processor -- start 2023-07-20T01:21:39.333 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.050+0000 7f4e7caa0700 1 -- start start 2023-07-20T01:21:39.333 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.051+0000 7f4e7caa0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e78106a20 0x7f4e78106380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:39.334 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.051+0000 7f4e7caa0700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e781068c0 con 0x7f4e78106a20 2023-07-20T01:21:39.334 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.051+0000 7f4e7659c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e78106a20 0x7f4e78106380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:39.334 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.051+0000 7f4e7659c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e78106a20 0x7f4e78106380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42034/0 (socket says 172.21.15.60:42034) 2023-07-20T01:21:39.334 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.051+0000 7f4e7659c700 1 -- 172.21.15.60:0/1088332445 learned_addr learned my addr 172.21.15.60:0/1088332445 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:39.334 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.051+0000 7f4e7659c700 1 -- 172.21.15.60:0/1088332445 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e68009550 con 0x7f4e78106a20 2023-07-20T01:21:39.334 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.051+0000 7f4e7659c700 1 --2- 172.21.15.60:0/1088332445 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e78106a20 0x7f4e78106380 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4e680118f0 tx=0x7f4e680119d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:39.334 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.051+0000 7f4e677fe700 1 -- 172.21.15.60:0/1088332445 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4e680036a0 con 0x7f4e78106a20 2023-07-20T01:21:39.335 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.051+0000 7f4e677fe700 1 -- 172.21.15.60:0/1088332445 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4e6801b440 con 0x7f4e78106a20 2023-07-20T01:21:39.335 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.051+0000 7f4e7caa0700 1 -- 172.21.15.60:0/1088332445 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e781049d0 con 0x7f4e78106a20 2023-07-20T01:21:39.335 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.051+0000 7f4e677fe700 1 -- 172.21.15.60:0/1088332445 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4e680193f0 con 0x7f4e78106a20 2023-07-20T01:21:39.335 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.051+0000 7f4e7caa0700 1 -- 172.21.15.60:0/1088332445 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e78104df0 con 0x7f4e78106a20 2023-07-20T01:21:39.335 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.053+0000 7f4e677fe700 1 -- 172.21.15.60:0/1088332445 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f4e6801b5b0 con 0x7f4e78106a20 2023-07-20T01:21:39.335 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.053+0000 7f4e677fe700 1 --2- 172.21.15.60:0/1088332445 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f4e600387b0 0x7f4e6003ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:39.335 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.053+0000 7f4e677fe700 1 -- 172.21.15.60:0/1088332445 --> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e6003b380 con 0x7f4e600387b0 2023-07-20T01:21:39.335 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.053+0000 7f4e677fe700 1 -- 172.21.15.60:0/1088332445 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f4e6804c5e0 con 0x7f4e78106a20 2023-07-20T01:21:39.336 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.053+0000 7f4e75d9b700 1 -- 172.21.15.60:0/1088332445 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f4e600387b0 msgr2=0x7f4e6003ac70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/712953724 2023-07-20T01:21:39.336 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.053+0000 7f4e75d9b700 1 --2- 172.21.15.60:0/1088332445 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f4e600387b0 0x7f4e6003ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T01:21:39.336 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.254+0000 7f4e75d9b700 1 -- 172.21.15.60:0/1088332445 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f4e600387b0 msgr2=0x7f4e6003ac70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/712953724 2023-07-20T01:21:39.336 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.254+0000 7f4e75d9b700 1 --2- 172.21.15.60:0/1088332445 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f4e600387b0 0x7f4e6003ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-20T01:21:39.336 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.655+0000 7f4e75d9b700 1 -- 172.21.15.60:0/1088332445 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f4e600387b0 msgr2=0x7f4e6003ac70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/712953724 2023-07-20T01:21:39.336 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:32.655+0000 7f4e75d9b700 1 --2- 172.21.15.60:0/1088332445 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f4e600387b0 0x7f4e6003ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-20T01:21:39.336 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:33.456+0000 7f4e75d9b700 1 -- 172.21.15.60:0/1088332445 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f4e600387b0 msgr2=0x7f4e6003ac70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/712953724 2023-07-20T01:21:39.337 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:33.456+0000 7f4e75d9b700 1 --2- 172.21.15.60:0/1088332445 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f4e600387b0 0x7f4e6003ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-20T01:21:39.337 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:35.057+0000 7f4e75d9b700 1 -- 172.21.15.60:0/1088332445 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f4e600387b0 msgr2=0x7f4e6003ac70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/712953724 2023-07-20T01:21:39.337 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:35.057+0000 7f4e75d9b700 1 --2- 172.21.15.60:0/1088332445 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f4e600387b0 0x7f4e6003ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-07-20T01:21:39.337 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:37.805+0000 7f4e677fe700 1 -- 172.21.15.60:0/1088332445 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mgrmap(e 10) v1 ==== 45263+0+0 (secure 0 0 0) 0x7f4e6801b900 con 0x7f4e78106a20 2023-07-20T01:21:39.337 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:37.805+0000 7f4e677fe700 1 -- 172.21.15.60:0/1088332445 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f4e600387b0 msgr2=0x7f4e6003ac70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T01:21:39.337 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:37.805+0000 7f4e677fe700 1 --2- 172.21.15.60:0/1088332445 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f4e600387b0 0x7f4e6003ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:39.337 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:38.806+0000 7f4e677fe700 1 -- 172.21.15.60:0/1088332445 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7f4e6804e130 con 0x7f4e78106a20 2023-07-20T01:21:39.337 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:38.807+0000 7f4e677fe700 1 --2- 172.21.15.60:0/1088332445 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f4e6003bfd0 0x7f4e6003e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:39.338 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:38.807+0000 7f4e677fe700 1 -- 172.21.15.60:0/1088332445 --> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e6003b380 con 0x7f4e6003bfd0 2023-07-20T01:21:39.338 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:38.807+0000 7f4e75d9b700 1 --2- 172.21.15.60:0/1088332445 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f4e6003bfd0 0x7f4e6003e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:39.338 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:38.808+0000 7f4e75d9b700 1 --2- 172.21.15.60:0/1088332445 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f4e6003bfd0 0x7f4e6003e3c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c003a10 tx=0x7f4e6c009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:39.338 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:38.809+0000 7f4e677fe700 1 -- 172.21.15.60:0/1088332445 <== mgr.14152 v2:172.21.15.60:6800/1961833180 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f4e6003b380 con 0x7f4e6003bfd0 2023-07-20T01:21:39.338 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:38.819+0000 7f4e7caa0700 1 -- 172.21.15.60:0/1088332445 --> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f4e78105f20 con 0x7f4e6003bfd0 2023-07-20T01:21:39.338 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:38.819+0000 7f4e677fe700 1 -- 172.21.15.60:0/1088332445 <== mgr.14152 v2:172.21.15.60:6800/1961833180 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f4e78105f20 con 0x7f4e6003bfd0 2023-07-20T01:21:39.339 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:38.820+0000 7f4e7caa0700 1 -- 172.21.15.60:0/1088332445 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f4e6003bfd0 msgr2=0x7f4e6003e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:39.339 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:38.820+0000 7f4e7caa0700 1 --2- 172.21.15.60:0/1088332445 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f4e6003bfd0 0x7f4e6003e3c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c003a10 tx=0x7f4e6c009270 comp rx=0 tx=0).stop 2023-07-20T01:21:39.339 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:38.820+0000 7f4e7caa0700 1 -- 172.21.15.60:0/1088332445 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e78106a20 msgr2=0x7f4e78106380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:39.339 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:38.820+0000 7f4e7caa0700 1 --2- 172.21.15.60:0/1088332445 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e78106a20 0x7f4e78106380 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4e680118f0 tx=0x7f4e680119d0 comp rx=0 tx=0).stop 2023-07-20T01:21:39.339 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:38.820+0000 7f4e7caa0700 1 -- 172.21.15.60:0/1088332445 shutdown_connections 2023-07-20T01:21:39.339 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:38.820+0000 7f4e7caa0700 1 --2- 172.21.15.60:0/1088332445 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f4e6003bfd0 0x7f4e6003e3c0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:39.339 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:38.820+0000 7f4e7caa0700 1 --2- 172.21.15.60:0/1088332445 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e78106a20 0x7f4e78106380 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:39.340 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:38.820+0000 7f4e7caa0700 1 --2- 172.21.15.60:0/1088332445 >> [v2:172.21.15.60:6800/712953724,v1:172.21.15.60:6801/712953724] conn(0x7f4e600387b0 0x7f4e6003ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:39.340 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:38.820+0000 7f4e7caa0700 1 -- 172.21.15.60:0/1088332445 >> 172.21.15.60:0/1088332445 conn(0x7f4e78100490 msgr2=0x7f4e78101b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:39.340 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:38.820+0000 7f4e7caa0700 1 -- 172.21.15.60:0/1088332445 shutdown_connections 2023-07-20T01:21:39.340 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:38.820+0000 7f4e7caa0700 1 -- 172.21.15.60:0/1088332445 wait complete. 2023-07-20T01:21:39.340 INFO:teuthology.orchestra.run.smithi060.stdout:mgr epoch 9 is available 2023-07-20T01:21:39.340 INFO:teuthology.orchestra.run.smithi060.stdout:Generating a dashboard self-signed certificate... 2023-07-20T01:21:40.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:39 smithi060 ceph-mon[96524]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-20T01:21:40.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:39 smithi060 ceph-mon[96524]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-20T01:21:40.879 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-07-20T01:21:40.879 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.030+0000 7f219678b700 1 Processor -- start 2023-07-20T01:21:40.879 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.030+0000 7f219678b700 1 -- start start 2023-07-20T01:21:40.879 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.030+0000 7f219678b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2190106a60 0x7f2190108e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:40.880 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.030+0000 7f219678b700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2190073ec0 con 0x7f2190106a60 2023-07-20T01:21:40.880 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.030+0000 7f218ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2190106a60 0x7f2190108e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:40.880 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.030+0000 7f218ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2190106a60 0x7f2190108e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42090/0 (socket says 172.21.15.60:42090) 2023-07-20T01:21:40.880 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.030+0000 7f218ffff700 1 -- 172.21.15.60:0/2270585318 learned_addr learned my addr 172.21.15.60:0/2270585318 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:40.880 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.031+0000 7f218ffff700 1 -- 172.21.15.60:0/2270585318 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2190109390 con 0x7f2190106a60 2023-07-20T01:21:40.880 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.031+0000 7f218ffff700 1 --2- 172.21.15.60:0/2270585318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2190106a60 0x7f2190108e50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f21800098a0 tx=0x7f2180009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0308b39254a2193 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:40.881 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.031+0000 7f218effd700 1 -- 172.21.15.60:0/2270585318 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2180004160 con 0x7f2190106a60 2023-07-20T01:21:40.881 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.031+0000 7f218effd700 1 -- 172.21.15.60:0/2270585318 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f21800042c0 con 0x7f2190106a60 2023-07-20T01:21:40.881 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.031+0000 7f218effd700 1 -- 172.21.15.60:0/2270585318 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f21800118e0 con 0x7f2190106a60 2023-07-20T01:21:40.881 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.032+0000 7f219678b700 1 -- 172.21.15.60:0/2270585318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2190106a60 msgr2=0x7f2190108e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:40.881 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.032+0000 7f219678b700 1 --2- 172.21.15.60:0/2270585318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2190106a60 0x7f2190108e50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f21800098a0 tx=0x7f2180009bb0 comp rx=0 tx=0).stop 2023-07-20T01:21:40.881 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.032+0000 7f219678b700 1 -- 172.21.15.60:0/2270585318 shutdown_connections 2023-07-20T01:21:40.881 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.032+0000 7f219678b700 1 --2- 172.21.15.60:0/2270585318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2190106a60 0x7f2190108e50 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:40.882 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.032+0000 7f219678b700 1 -- 172.21.15.60:0/2270585318 >> 172.21.15.60:0/2270585318 conn(0x7f2190100490 msgr2=0x7f21901028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:40.882 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.032+0000 7f219678b700 1 -- 172.21.15.60:0/2270585318 shutdown_connections 2023-07-20T01:21:40.882 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.032+0000 7f219678b700 1 -- 172.21.15.60:0/2270585318 wait complete. 2023-07-20T01:21:40.882 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.033+0000 7f219678b700 1 Processor -- start 2023-07-20T01:21:40.882 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.033+0000 7f219678b700 1 -- start start 2023-07-20T01:21:40.882 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.033+0000 7f219678b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2190106a60 0x7f2190199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:40.883 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.033+0000 7f219678b700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f219019a1c0 con 0x7f2190106a60 2023-07-20T01:21:40.883 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.033+0000 7f218ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2190106a60 0x7f2190199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:40.883 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.033+0000 7f218ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2190106a60 0x7f2190199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42092/0 (socket says 172.21.15.60:42092) 2023-07-20T01:21:40.883 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.033+0000 7f218ffff700 1 -- 172.21.15.60:0/478134577 learned_addr learned my addr 172.21.15.60:0/478134577 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:40.883 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.034+0000 7f218ffff700 1 -- 172.21.15.60:0/478134577 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2180009550 con 0x7f2190106a60 2023-07-20T01:21:40.883 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.034+0000 7f218ffff700 1 --2- 172.21.15.60:0/478134577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2190106a60 0x7f2190199c80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f21800118f0 tx=0x7f21800119d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:40.883 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.034+0000 7f218d7fa700 1 -- 172.21.15.60:0/478134577 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f21800036a0 con 0x7f2190106a60 2023-07-20T01:21:40.884 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.034+0000 7f218d7fa700 1 -- 172.21.15.60:0/478134577 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f218001b440 con 0x7f2190106a60 2023-07-20T01:21:40.884 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.034+0000 7f219678b700 1 -- 172.21.15.60:0/478134577 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f219019a3c0 con 0x7f2190106a60 2023-07-20T01:21:40.884 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.034+0000 7f218d7fa700 1 -- 172.21.15.60:0/478134577 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f21800193f0 con 0x7f2190106a60 2023-07-20T01:21:40.884 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.034+0000 7f219678b700 1 -- 172.21.15.60:0/478134577 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f219019a7e0 con 0x7f2190106a60 2023-07-20T01:21:40.884 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.035+0000 7f218d7fa700 1 -- 172.21.15.60:0/478134577 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f21800196b0 con 0x7f2190106a60 2023-07-20T01:21:40.884 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.035+0000 7f218d7fa700 1 --2- 172.21.15.60:0/478134577 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f2178038760 0x7f217803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:40.884 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.035+0000 7f218d7fa700 1 -- 172.21.15.60:0/478134577 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f218004c6f0 con 0x7f2190106a60 2023-07-20T01:21:40.885 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.035+0000 7f218f7fe700 1 --2- 172.21.15.60:0/478134577 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f2178038760 0x7f217803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:40.885 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.035+0000 7f219678b700 1 -- 172.21.15.60:0/478134577 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2190061da0 con 0x7f2190106a60 2023-07-20T01:21:40.885 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.036+0000 7f218f7fe700 1 --2- 172.21.15.60:0/478134577 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f2178038760 0x7f217803ac20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2184006fd0 tx=0x7f2184006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:40.885 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.040+0000 7f218d7fa700 1 -- 172.21.15.60:0/478134577 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f218001f020 con 0x7f2190106a60 2023-07-20T01:21:40.885 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.409+0000 7f219678b700 1 -- 172.21.15.60:0/478134577 --> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f219019d330 con 0x7f2178038760 2023-07-20T01:21:40.885 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.480+0000 7f218d7fa700 1 -- 172.21.15.60:0/478134577 <== mgr.14152 v2:172.21.15.60:6800/1961833180 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f219019d330 con 0x7f2178038760 2023-07-20T01:21:40.885 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.482+0000 7f219678b700 1 -- 172.21.15.60:0/478134577 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f2178038760 msgr2=0x7f217803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:40.886 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.482+0000 7f219678b700 1 --2- 172.21.15.60:0/478134577 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f2178038760 0x7f217803ac20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2184006fd0 tx=0x7f2184006e40 comp rx=0 tx=0).stop 2023-07-20T01:21:40.886 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.482+0000 7f219678b700 1 -- 172.21.15.60:0/478134577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2190106a60 msgr2=0x7f2190199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:40.886 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.482+0000 7f219678b700 1 --2- 172.21.15.60:0/478134577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2190106a60 0x7f2190199c80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f21800118f0 tx=0x7f21800119d0 comp rx=0 tx=0).stop 2023-07-20T01:21:40.886 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.482+0000 7f219678b700 1 -- 172.21.15.60:0/478134577 shutdown_connections 2023-07-20T01:21:40.886 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.482+0000 7f219678b700 1 --2- 172.21.15.60:0/478134577 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f2178038760 0x7f217803ac20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:40.886 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.482+0000 7f219678b700 1 --2- 172.21.15.60:0/478134577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2190106a60 0x7f2190199c80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:40.886 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.482+0000 7f219678b700 1 -- 172.21.15.60:0/478134577 >> 172.21.15.60:0/478134577 conn(0x7f2190100490 msgr2=0x7f2190101b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:40.886 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.482+0000 7f219678b700 1 -- 172.21.15.60:0/478134577 shutdown_connections 2023-07-20T01:21:40.887 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:40.483+0000 7f219678b700 1 -- 172.21.15.60:0/478134577 wait complete. 2023-07-20T01:21:40.887 INFO:teuthology.orchestra.run.smithi060.stdout:Creating initial admin user... 2023-07-20T01:21:41.097 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:40 smithi060 ceph-mon[96524]: mgrmap e12: x(active, since 2s) 2023-07-20T01:21:41.098 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:40 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:21:41.098 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:40 smithi060 ceph-mon[96524]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:21:41.098 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:40 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:21:41.098 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:40 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:21:42.030 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:41 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:21:42.031 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:41 smithi060 ceph-mon[96524]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:21:42.727 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$DIP1zzluoflJuIbOXf9DNO1dsNgh3HQDpyAbxx8GqJYJ4WPaf.xwK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1689816102, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-07-20T01:21:42.728 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.363+0000 7fb238387700 1 Processor -- start 2023-07-20T01:21:42.728 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.363+0000 7fb238387700 1 -- start start 2023-07-20T01:21:42.728 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.363+0000 7fb238387700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb230106a20 0x7fb230108e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:42.728 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.363+0000 7fb238387700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb230079e50 con 0x7fb230106a20 2023-07-20T01:21:42.728 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.364+0000 7fb236123700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb230106a20 0x7fb230108e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:42.729 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.364+0000 7fb236123700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb230106a20 0x7fb230108e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42096/0 (socket says 172.21.15.60:42096) 2023-07-20T01:21:42.729 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.364+0000 7fb236123700 1 -- 172.21.15.60:0/780895709 learned_addr learned my addr 172.21.15.60:0/780895709 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:42.729 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.364+0000 7fb236123700 1 -- 172.21.15.60:0/780895709 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb230109350 con 0x7fb230106a20 2023-07-20T01:21:42.729 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.364+0000 7fb236123700 1 --2- 172.21.15.60:0/780895709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb230106a20 0x7fb230108e10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb22c0098a0 tx=0x7fb22c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b07569ba170d7ff5 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:42.729 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.364+0000 7fb235121700 1 -- 172.21.15.60:0/780895709 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb22c004160 con 0x7fb230106a20 2023-07-20T01:21:42.729 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.365+0000 7fb235121700 1 -- 172.21.15.60:0/780895709 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb22c0042c0 con 0x7fb230106a20 2023-07-20T01:21:42.729 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.365+0000 7fb235121700 1 -- 172.21.15.60:0/780895709 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb22c0118e0 con 0x7fb230106a20 2023-07-20T01:21:42.730 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.365+0000 7fb238387700 1 -- 172.21.15.60:0/780895709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb230106a20 msgr2=0x7fb230108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:42.730 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.365+0000 7fb238387700 1 --2- 172.21.15.60:0/780895709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb230106a20 0x7fb230108e10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb22c0098a0 tx=0x7fb22c009bb0 comp rx=0 tx=0).stop 2023-07-20T01:21:42.730 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.365+0000 7fb238387700 1 -- 172.21.15.60:0/780895709 shutdown_connections 2023-07-20T01:21:42.730 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.365+0000 7fb238387700 1 --2- 172.21.15.60:0/780895709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb230106a20 0x7fb230108e10 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:42.730 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.365+0000 7fb238387700 1 -- 172.21.15.60:0/780895709 >> 172.21.15.60:0/780895709 conn(0x7fb2301004b0 msgr2=0x7fb2301028d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:42.730 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.365+0000 7fb238387700 1 -- 172.21.15.60:0/780895709 shutdown_connections 2023-07-20T01:21:42.730 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.365+0000 7fb238387700 1 -- 172.21.15.60:0/780895709 wait complete. 2023-07-20T01:21:42.731 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.366+0000 7fb238387700 1 Processor -- start 2023-07-20T01:21:42.731 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.366+0000 7fb238387700 1 -- start start 2023-07-20T01:21:42.731 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.366+0000 7fb238387700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb230106a20 0x7fb230199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:42.731 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.366+0000 7fb238387700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23019a240 con 0x7fb230106a20 2023-07-20T01:21:42.731 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.366+0000 7fb236123700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb230106a20 0x7fb230199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:42.731 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.366+0000 7fb236123700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb230106a20 0x7fb230199d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42098/0 (socket says 172.21.15.60:42098) 2023-07-20T01:21:42.731 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.366+0000 7fb236123700 1 -- 172.21.15.60:0/94615099 learned_addr learned my addr 172.21.15.60:0/94615099 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:42.731 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.367+0000 7fb236123700 1 -- 172.21.15.60:0/94615099 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb22c009550 con 0x7fb230106a20 2023-07-20T01:21:42.732 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.367+0000 7fb236123700 1 --2- 172.21.15.60:0/94615099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb230106a20 0x7fb230199d00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb22c006af0 tx=0x7fb22c011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:42.732 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.367+0000 7fb2277fe700 1 -- 172.21.15.60:0/94615099 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb22c0037d0 con 0x7fb230106a20 2023-07-20T01:21:42.732 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.367+0000 7fb2277fe700 1 -- 172.21.15.60:0/94615099 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb22c01b440 con 0x7fb230106a20 2023-07-20T01:21:42.732 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.367+0000 7fb238387700 1 -- 172.21.15.60:0/94615099 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb23019a440 con 0x7fb230106a20 2023-07-20T01:21:42.732 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.367+0000 7fb2277fe700 1 -- 172.21.15.60:0/94615099 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb22c023400 con 0x7fb230106a20 2023-07-20T01:21:42.732 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.367+0000 7fb238387700 1 -- 172.21.15.60:0/94615099 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb23019a860 con 0x7fb230106a20 2023-07-20T01:21:42.732 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.368+0000 7fb2277fe700 1 -- 172.21.15.60:0/94615099 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7fb22c01b5b0 con 0x7fb230106a20 2023-07-20T01:21:42.733 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.368+0000 7fb2277fe700 1 --2- 172.21.15.60:0/94615099 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fb21c038760 0x7fb21c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:42.733 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.368+0000 7fb2277fe700 1 -- 172.21.15.60:0/94615099 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb22c04c860 con 0x7fb230106a20 2023-07-20T01:21:42.733 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.368+0000 7fb235922700 1 --2- 172.21.15.60:0/94615099 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fb21c038760 0x7fb21c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:42.733 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.368+0000 7fb238387700 1 -- 172.21.15.60:0/94615099 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb230193d50 con 0x7fb230106a20 2023-07-20T01:21:42.733 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.369+0000 7fb235922700 1 --2- 172.21.15.60:0/94615099 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fb21c038760 0x7fb21c03ac20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb220006fd0 tx=0x7fb220006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:42.733 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.373+0000 7fb2277fe700 1 -- 172.21.15.60:0/94615099 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb22c01bab0 con 0x7fb230106a20 2023-07-20T01:21:42.733 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:41.771+0000 7fb238387700 1 -- 172.21.15.60:0/94615099 --> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fb230061bc0 con 0x7fb21c038760 2023-07-20T01:21:42.733 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:42.019+0000 7fb2277fe700 1 -- 172.21.15.60:0/94615099 <== mgr.14152 v2:172.21.15.60:6800/1961833180 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fb230061bc0 con 0x7fb21c038760 2023-07-20T01:21:42.734 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:42.021+0000 7fb238387700 1 -- 172.21.15.60:0/94615099 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fb21c038760 msgr2=0x7fb21c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:42.734 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:42.021+0000 7fb238387700 1 --2- 172.21.15.60:0/94615099 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fb21c038760 0x7fb21c03ac20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb220006fd0 tx=0x7fb220006e40 comp rx=0 tx=0).stop 2023-07-20T01:21:42.734 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:42.022+0000 7fb238387700 1 -- 172.21.15.60:0/94615099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb230106a20 msgr2=0x7fb230199d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:42.734 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:42.022+0000 7fb238387700 1 --2- 172.21.15.60:0/94615099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb230106a20 0x7fb230199d00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb22c006af0 tx=0x7fb22c011ed0 comp rx=0 tx=0).stop 2023-07-20T01:21:42.734 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:42.022+0000 7fb238387700 1 -- 172.21.15.60:0/94615099 shutdown_connections 2023-07-20T01:21:42.734 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:42.022+0000 7fb238387700 1 --2- 172.21.15.60:0/94615099 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fb21c038760 0x7fb21c03ac20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:42.734 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:42.022+0000 7fb238387700 1 --2- 172.21.15.60:0/94615099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb230106a20 0x7fb230199d00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:42.735 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:42.022+0000 7fb238387700 1 -- 172.21.15.60:0/94615099 >> 172.21.15.60:0/94615099 conn(0x7fb2301004b0 msgr2=0x7fb2301028a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:42.735 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:42.022+0000 7fb238387700 1 -- 172.21.15.60:0/94615099 shutdown_connections 2023-07-20T01:21:42.735 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:42.022+0000 7fb238387700 1 -- 172.21.15.60:0/94615099 wait complete. 2023-07-20T01:21:42.735 INFO:teuthology.orchestra.run.smithi060.stdout:Fetching dashboard port number... 2023-07-20T01:21:43.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:43 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:21:43.978 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 8443 2023-07-20T01:21:43.979 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.230+0000 7f560370d700 1 Processor -- start 2023-07-20T01:21:43.979 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.230+0000 7f560370d700 1 -- start start 2023-07-20T01:21:43.979 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.231+0000 7f560370d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55fc105f90 0x7f55fc1063b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:43.979 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.231+0000 7f560370d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc106980 con 0x7f55fc105f90 2023-07-20T01:21:43.979 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.231+0000 7f56014a9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55fc105f90 0x7f55fc1063b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:43.980 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.231+0000 7f56014a9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55fc105f90 0x7f55fc1063b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42106/0 (socket says 172.21.15.60:42106) 2023-07-20T01:21:43.980 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.231+0000 7f56014a9700 1 -- 172.21.15.60:0/2067783181 learned_addr learned my addr 172.21.15.60:0/2067783181 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:43.980 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.231+0000 7f56014a9700 1 -- 172.21.15.60:0/2067783181 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55fc107190 con 0x7f55fc105f90 2023-07-20T01:21:43.980 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.231+0000 7f56014a9700 1 --2- 172.21.15.60:0/2067783181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55fc105f90 0x7f55fc1063b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f55ec0098a0 tx=0x7f55ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b234f5e39d25423f server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:43.980 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.231+0000 7f55f3fff700 1 -- 172.21.15.60:0/2067783181 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f55ec004160 con 0x7f55fc105f90 2023-07-20T01:21:43.980 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.231+0000 7f55f3fff700 1 -- 172.21.15.60:0/2067783181 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f55ec0042c0 con 0x7f55fc105f90 2023-07-20T01:21:43.980 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.231+0000 7f55f3fff700 1 -- 172.21.15.60:0/2067783181 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f55ec0118e0 con 0x7f55fc105f90 2023-07-20T01:21:43.981 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.232+0000 7f560370d700 1 -- 172.21.15.60:0/2067783181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55fc105f90 msgr2=0x7f55fc1063b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:43.981 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.232+0000 7f560370d700 1 --2- 172.21.15.60:0/2067783181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55fc105f90 0x7f55fc1063b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f55ec0098a0 tx=0x7f55ec009bb0 comp rx=0 tx=0).stop 2023-07-20T01:21:43.981 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.232+0000 7f560370d700 1 -- 172.21.15.60:0/2067783181 shutdown_connections 2023-07-20T01:21:43.981 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.232+0000 7f560370d700 1 --2- 172.21.15.60:0/2067783181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55fc105f90 0x7f55fc1063b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:43.981 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.232+0000 7f560370d700 1 -- 172.21.15.60:0/2067783181 >> 172.21.15.60:0/2067783181 conn(0x7f55fc101590 msgr2=0x7f55fc1039b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:43.981 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.232+0000 7f560370d700 1 -- 172.21.15.60:0/2067783181 shutdown_connections 2023-07-20T01:21:43.981 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.232+0000 7f560370d700 1 -- 172.21.15.60:0/2067783181 wait complete. 2023-07-20T01:21:43.982 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.233+0000 7f560370d700 1 Processor -- start 2023-07-20T01:21:43.982 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.233+0000 7f560370d700 1 -- start start 2023-07-20T01:21:43.982 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.233+0000 7f560370d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55fc105f90 0x7f55fc199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:43.982 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.233+0000 7f560370d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc19a1a0 con 0x7f55fc105f90 2023-07-20T01:21:43.982 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.233+0000 7f56014a9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55fc105f90 0x7f55fc199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:43.982 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.233+0000 7f56014a9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55fc105f90 0x7f55fc199c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42108/0 (socket says 172.21.15.60:42108) 2023-07-20T01:21:43.982 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.233+0000 7f56014a9700 1 -- 172.21.15.60:0/385144367 learned_addr learned my addr 172.21.15.60:0/385144367 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:43.982 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.233+0000 7f56014a9700 1 -- 172.21.15.60:0/385144367 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55ec009550 con 0x7f55fc105f90 2023-07-20T01:21:43.983 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.234+0000 7f56014a9700 1 --2- 172.21.15.60:0/385144367 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55fc105f90 0x7f55fc199c60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f55ec000c00 tx=0x7f55ec0119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:43.983 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.234+0000 7f55f27fc700 1 -- 172.21.15.60:0/385144367 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f55ec0036a0 con 0x7f55fc105f90 2023-07-20T01:21:43.983 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.234+0000 7f55f27fc700 1 -- 172.21.15.60:0/385144367 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f55ec01b440 con 0x7f55fc105f90 2023-07-20T01:21:43.983 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.234+0000 7f560370d700 1 -- 172.21.15.60:0/385144367 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55fc19a3a0 con 0x7f55fc105f90 2023-07-20T01:21:43.983 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.234+0000 7f55f27fc700 1 -- 172.21.15.60:0/385144367 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f55ec01a3d0 con 0x7f55fc105f90 2023-07-20T01:21:43.983 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.234+0000 7f560370d700 1 -- 172.21.15.60:0/385144367 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55fc19a7c0 con 0x7f55fc105f90 2023-07-20T01:21:43.984 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.235+0000 7f55f27fc700 1 -- 172.21.15.60:0/385144367 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f55ec01a640 con 0x7f55fc105f90 2023-07-20T01:21:43.984 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.235+0000 7f55f27fc700 1 --2- 172.21.15.60:0/385144367 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f55e80387b0 0x7f55e803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:43.984 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.235+0000 7f55f27fc700 1 -- 172.21.15.60:0/385144367 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f55ec04c8a0 con 0x7f55fc105f90 2023-07-20T01:21:43.984 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.235+0000 7f5600ca8700 1 --2- 172.21.15.60:0/385144367 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f55e80387b0 0x7f55e803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:43.984 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.235+0000 7f560370d700 1 -- 172.21.15.60:0/385144367 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55fc061da0 con 0x7f55fc105f90 2023-07-20T01:21:43.985 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.235+0000 7f5600ca8700 1 --2- 172.21.15.60:0/385144367 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f55e80387b0 0x7f55e803ac70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f55f8006fd0 tx=0x7f55f8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:43.985 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.240+0000 7f55f27fc700 1 -- 172.21.15.60:0/385144367 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f55ec0239a0 con 0x7f55fc105f90 2023-07-20T01:21:43.985 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.595+0000 7f560370d700 1 -- 172.21.15.60:0/385144367 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f55fc10a390 con 0x7f55fc105f90 2023-07-20T01:21:43.985 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.596+0000 7f55f27fc700 1 -- 172.21.15.60:0/385144367 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f55ec0233b0 con 0x7f55fc105f90 2023-07-20T01:21:43.985 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.598+0000 7f560370d700 1 -- 172.21.15.60:0/385144367 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f55e80387b0 msgr2=0x7f55e803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:43.985 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.598+0000 7f560370d700 1 --2- 172.21.15.60:0/385144367 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f55e80387b0 0x7f55e803ac70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f55f8006fd0 tx=0x7f55f8006e40 comp rx=0 tx=0).stop 2023-07-20T01:21:43.985 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.598+0000 7f560370d700 1 -- 172.21.15.60:0/385144367 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55fc105f90 msgr2=0x7f55fc199c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:43.986 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.598+0000 7f560370d700 1 --2- 172.21.15.60:0/385144367 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55fc105f90 0x7f55fc199c60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f55ec000c00 tx=0x7f55ec0119a0 comp rx=0 tx=0).stop 2023-07-20T01:21:43.986 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.598+0000 7f560370d700 1 -- 172.21.15.60:0/385144367 shutdown_connections 2023-07-20T01:21:43.986 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.598+0000 7f560370d700 1 --2- 172.21.15.60:0/385144367 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f55e80387b0 0x7f55e803ac70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:43.986 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.598+0000 7f560370d700 1 --2- 172.21.15.60:0/385144367 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55fc105f90 0x7f55fc199c60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:43.986 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.598+0000 7f560370d700 1 -- 172.21.15.60:0/385144367 >> 172.21.15.60:0/385144367 conn(0x7f55fc101590 msgr2=0x7f55fc102cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:43.986 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.598+0000 7f560370d700 1 -- 172.21.15.60:0/385144367 shutdown_connections 2023-07-20T01:21:43.986 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:43.598+0000 7f560370d700 1 -- 172.21.15.60:0/385144367 wait complete. 2023-07-20T01:21:43.987 INFO:teuthology.orchestra.run.smithi060.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-20T01:21:43.987 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stdout disabled 2023-07-20T01:21:43.992 INFO:teuthology.orchestra.run.smithi060.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-20T01:21:43.992 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stdout inactive 2023-07-20T01:21:43.992 INFO:teuthology.orchestra.run.smithi060.stdout:firewalld.service is not enabled 2023-07-20T01:21:43.992 INFO:teuthology.orchestra.run.smithi060.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-07-20T01:21:43.993 INFO:teuthology.orchestra.run.smithi060.stdout:Ceph Dashboard is now available at: 2023-07-20T01:21:43.994 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:21:43.994 INFO:teuthology.orchestra.run.smithi060.stdout: URL: https://smithi060:8443/ 2023-07-20T01:21:43.994 INFO:teuthology.orchestra.run.smithi060.stdout: User: admin 2023-07-20T01:21:43.994 INFO:teuthology.orchestra.run.smithi060.stdout: Password: lj4vg8bsmu 2023-07-20T01:21:43.994 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:21:43.995 INFO:teuthology.orchestra.run.smithi060.stdout:Saving cluster configuration to /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/config directory 2023-07-20T01:21:43.995 INFO:teuthology.orchestra.run.smithi060.stdout:Enabling autotune for osd_memory_target 2023-07-20T01:21:44.199 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:44 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/385144367' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-07-20T01:21:45.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:45 smithi060 ceph-mon[96524]: mgrmap e13: x(active, since 6s) 2023-07-20T01:21:45.241 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.481+0000 7f807f875700 1 Processor -- start 2023-07-20T01:21:45.241 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.481+0000 7f807f875700 1 -- start start 2023-07-20T01:21:45.241 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.482+0000 7f807f875700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8078106a70 0x7f8078108e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:45.241 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.482+0000 7f807f875700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8078073ec0 con 0x7f8078106a70 2023-07-20T01:21:45.241 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.482+0000 7f807d611700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8078106a70 0x7f8078108e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:45.242 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.482+0000 7f807d611700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8078106a70 0x7f8078108e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42112/0 (socket says 172.21.15.60:42112) 2023-07-20T01:21:45.242 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.482+0000 7f807d611700 1 -- 172.21.15.60:0/2438586870 learned_addr learned my addr 172.21.15.60:0/2438586870 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:45.242 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.482+0000 7f807d611700 1 -- 172.21.15.60:0/2438586870 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80781093a0 con 0x7f8078106a70 2023-07-20T01:21:45.242 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.482+0000 7f807d611700 1 --2- 172.21.15.60:0/2438586870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8078106a70 0x7f8078108e60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f80740098a0 tx=0x7f8074009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=836f8b499287321c server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:45.242 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.482+0000 7f806ffff700 1 -- 172.21.15.60:0/2438586870 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8074004160 con 0x7f8078106a70 2023-07-20T01:21:45.242 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.482+0000 7f806ffff700 1 -- 172.21.15.60:0/2438586870 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f80740042c0 con 0x7f8078106a70 2023-07-20T01:21:45.242 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.482+0000 7f806ffff700 1 -- 172.21.15.60:0/2438586870 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f80740118e0 con 0x7f8078106a70 2023-07-20T01:21:45.243 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.483+0000 7f807f875700 1 -- 172.21.15.60:0/2438586870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8078106a70 msgr2=0x7f8078108e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:45.243 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.483+0000 7f807f875700 1 --2- 172.21.15.60:0/2438586870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8078106a70 0x7f8078108e60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f80740098a0 tx=0x7f8074009bb0 comp rx=0 tx=0).stop 2023-07-20T01:21:45.243 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.483+0000 7f807f875700 1 -- 172.21.15.60:0/2438586870 shutdown_connections 2023-07-20T01:21:45.243 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.483+0000 7f807f875700 1 --2- 172.21.15.60:0/2438586870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8078106a70 0x7f8078108e60 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:45.243 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.483+0000 7f807f875700 1 -- 172.21.15.60:0/2438586870 >> 172.21.15.60:0/2438586870 conn(0x7f80781004a0 msgr2=0x7f80781028c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:45.243 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.483+0000 7f807f875700 1 -- 172.21.15.60:0/2438586870 shutdown_connections 2023-07-20T01:21:45.243 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.483+0000 7f807f875700 1 -- 172.21.15.60:0/2438586870 wait complete. 2023-07-20T01:21:45.244 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.484+0000 7f807f875700 1 Processor -- start 2023-07-20T01:21:45.244 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.484+0000 7f807f875700 1 -- start start 2023-07-20T01:21:45.244 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.484+0000 7f807f875700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8078106a70 0x7f8078199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:45.244 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.484+0000 7f807f875700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807819a1d0 con 0x7f8078106a70 2023-07-20T01:21:45.244 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.484+0000 7f807d611700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8078106a70 0x7f8078199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:45.244 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.484+0000 7f807d611700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8078106a70 0x7f8078199c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42114/0 (socket says 172.21.15.60:42114) 2023-07-20T01:21:45.244 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.484+0000 7f807d611700 1 -- 172.21.15.60:0/2208586213 learned_addr learned my addr 172.21.15.60:0/2208586213 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:45.244 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.484+0000 7f807d611700 1 -- 172.21.15.60:0/2208586213 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8074009550 con 0x7f8078106a70 2023-07-20T01:21:45.245 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.485+0000 7f807d611700 1 --2- 172.21.15.60:0/2208586213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8078106a70 0x7f8078199c90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8074006af0 tx=0x7f8074011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:45.245 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.485+0000 7f806e7fc700 1 -- 172.21.15.60:0/2208586213 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f80740037d0 con 0x7f8078106a70 2023-07-20T01:21:45.245 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.485+0000 7f806e7fc700 1 -- 172.21.15.60:0/2208586213 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f807401b440 con 0x7f8078106a70 2023-07-20T01:21:45.245 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.485+0000 7f806e7fc700 1 -- 172.21.15.60:0/2208586213 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8074023400 con 0x7f8078106a70 2023-07-20T01:21:45.245 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.485+0000 7f807f875700 1 -- 172.21.15.60:0/2208586213 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f807819a3d0 con 0x7f8078106a70 2023-07-20T01:21:45.245 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.485+0000 7f807f875700 1 -- 172.21.15.60:0/2208586213 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f807819a7f0 con 0x7f8078106a70 2023-07-20T01:21:45.245 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.486+0000 7f807f875700 1 -- 172.21.15.60:0/2208586213 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8078061da0 con 0x7f8078106a70 2023-07-20T01:21:45.246 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.486+0000 7f806e7fc700 1 -- 172.21.15.60:0/2208586213 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f807401b5b0 con 0x7f8078106a70 2023-07-20T01:21:45.246 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.487+0000 7f806e7fc700 1 --2- 172.21.15.60:0/2208586213 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f8064038800 0x7f806403acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:45.246 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.487+0000 7f806e7fc700 1 -- 172.21.15.60:0/2208586213 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f807404cd90 con 0x7f8078106a70 2023-07-20T01:21:45.246 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.487+0000 7f807ce10700 1 --2- 172.21.15.60:0/2208586213 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f8064038800 0x7f806403acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:45.246 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.487+0000 7f807ce10700 1 --2- 172.21.15.60:0/2208586213 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f8064038800 0x7f806403acc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8068006fd0 tx=0x7f8068006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:45.246 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.491+0000 7f806e7fc700 1 -- 172.21.15.60:0/2208586213 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f807401b8d0 con 0x7f8078106a70 2023-07-20T01:21:45.246 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.847+0000 7f807f875700 1 -- 172.21.15.60:0/2208586213 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f807819d640 con 0x7f8078106a70 2023-07-20T01:21:45.247 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.848+0000 7f806e7fc700 1 -- 172.21.15.60:0/2208586213 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f807402bb40 con 0x7f8078106a70 2023-07-20T01:21:45.247 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.850+0000 7f807f875700 1 -- 172.21.15.60:0/2208586213 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f8064038800 msgr2=0x7f806403acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:45.247 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.850+0000 7f807f875700 1 --2- 172.21.15.60:0/2208586213 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f8064038800 0x7f806403acc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8068006fd0 tx=0x7f8068006e40 comp rx=0 tx=0).stop 2023-07-20T01:21:45.247 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.850+0000 7f807f875700 1 -- 172.21.15.60:0/2208586213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8078106a70 msgr2=0x7f8078199c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:45.247 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.850+0000 7f807f875700 1 --2- 172.21.15.60:0/2208586213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8078106a70 0x7f8078199c90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8074006af0 tx=0x7f8074011ed0 comp rx=0 tx=0).stop 2023-07-20T01:21:45.247 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.850+0000 7f807f875700 1 -- 172.21.15.60:0/2208586213 shutdown_connections 2023-07-20T01:21:45.247 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.850+0000 7f807f875700 1 --2- 172.21.15.60:0/2208586213 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f8064038800 0x7f806403acc0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:45.248 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.850+0000 7f807f875700 1 --2- 172.21.15.60:0/2208586213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8078106a70 0x7f8078199c90 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:45.248 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.850+0000 7f807f875700 1 -- 172.21.15.60:0/2208586213 >> 172.21.15.60:0/2208586213 conn(0x7f80781004a0 msgr2=0x7f8078101bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:45.248 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.850+0000 7f807f875700 1 -- 172.21.15.60:0/2208586213 shutdown_connections 2023-07-20T01:21:45.248 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:44.850+0000 7f807f875700 1 -- 172.21.15.60:0/2208586213 wait complete. 2023-07-20T01:21:46.684 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.750+0000 7fc70e6ed700 1 Processor -- start 2023-07-20T01:21:46.685 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.750+0000 7fc70e6ed700 1 -- start start 2023-07-20T01:21:46.685 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.751+0000 7fc70e6ed700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc708106a40 0x7fc708108e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:46.685 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.751+0000 7fc70e6ed700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc708079e10 con 0x7fc708106a40 2023-07-20T01:21:46.685 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.751+0000 7fc707fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc708106a40 0x7fc708108e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:46.685 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.751+0000 7fc707fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc708106a40 0x7fc708108e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42118/0 (socket says 172.21.15.60:42118) 2023-07-20T01:21:46.685 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.751+0000 7fc707fff700 1 -- 172.21.15.60:0/1930653048 learned_addr learned my addr 172.21.15.60:0/1930653048 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:46.686 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.751+0000 7fc707fff700 1 -- 172.21.15.60:0/1930653048 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc708109370 con 0x7fc708106a40 2023-07-20T01:21:46.686 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.751+0000 7fc707fff700 1 --2- 172.21.15.60:0/1930653048 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc708106a40 0x7fc708108e30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc0098a0 tx=0x7fc6fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77969c59a38d02ee server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:46.686 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.751+0000 7fc706ffd700 1 -- 172.21.15.60:0/1930653048 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc6fc004160 con 0x7fc708106a40 2023-07-20T01:21:46.686 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.751+0000 7fc706ffd700 1 -- 172.21.15.60:0/1930653048 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc6fc0042c0 con 0x7fc708106a40 2023-07-20T01:21:46.686 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.751+0000 7fc706ffd700 1 -- 172.21.15.60:0/1930653048 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc6fc0118e0 con 0x7fc708106a40 2023-07-20T01:21:46.686 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.752+0000 7fc70e6ed700 1 -- 172.21.15.60:0/1930653048 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc708106a40 msgr2=0x7fc708108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:46.686 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.752+0000 7fc70e6ed700 1 --2- 172.21.15.60:0/1930653048 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc708106a40 0x7fc708108e30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc0098a0 tx=0x7fc6fc009bb0 comp rx=0 tx=0).stop 2023-07-20T01:21:46.687 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.752+0000 7fc70e6ed700 1 -- 172.21.15.60:0/1930653048 shutdown_connections 2023-07-20T01:21:46.687 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.752+0000 7fc70e6ed700 1 --2- 172.21.15.60:0/1930653048 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc708106a40 0x7fc708108e30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:46.687 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.752+0000 7fc70e6ed700 1 -- 172.21.15.60:0/1930653048 >> 172.21.15.60:0/1930653048 conn(0x7fc708100410 msgr2=0x7fc708102870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:46.687 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.752+0000 7fc70e6ed700 1 -- 172.21.15.60:0/1930653048 shutdown_connections 2023-07-20T01:21:46.687 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.752+0000 7fc70e6ed700 1 -- 172.21.15.60:0/1930653048 wait complete. 2023-07-20T01:21:46.687 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.753+0000 7fc70e6ed700 1 Processor -- start 2023-07-20T01:21:46.687 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.753+0000 7fc70e6ed700 1 -- start start 2023-07-20T01:21:46.688 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.753+0000 7fc70e6ed700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc708106a40 0x7fc7081063a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:46.688 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.753+0000 7fc70e6ed700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7081068e0 con 0x7fc708106a40 2023-07-20T01:21:46.688 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.753+0000 7fc707fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc708106a40 0x7fc7081063a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:46.688 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.753+0000 7fc707fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc708106a40 0x7fc7081063a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42120/0 (socket says 172.21.15.60:42120) 2023-07-20T01:21:46.688 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.753+0000 7fc707fff700 1 -- 172.21.15.60:0/2296098992 learned_addr learned my addr 172.21.15.60:0/2296098992 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:46.688 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.753+0000 7fc707fff700 1 -- 172.21.15.60:0/2296098992 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6fc009550 con 0x7fc708106a40 2023-07-20T01:21:46.688 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.754+0000 7fc707fff700 1 --2- 172.21.15.60:0/2296098992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc708106a40 0x7fc7081063a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc006af0 tx=0x7fc6fc011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:46.689 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.754+0000 7fc7057fa700 1 -- 172.21.15.60:0/2296098992 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc6fc0037d0 con 0x7fc708106a40 2023-07-20T01:21:46.689 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.754+0000 7fc70e6ed700 1 -- 172.21.15.60:0/2296098992 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7081049f0 con 0x7fc708106a40 2023-07-20T01:21:46.689 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.754+0000 7fc7057fa700 1 -- 172.21.15.60:0/2296098992 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc6fc01b440 con 0x7fc708106a40 2023-07-20T01:21:46.689 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.754+0000 7fc7057fa700 1 -- 172.21.15.60:0/2296098992 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc6fc0193d0 con 0x7fc708106a40 2023-07-20T01:21:46.689 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.754+0000 7fc70e6ed700 1 -- 172.21.15.60:0/2296098992 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc708104e10 con 0x7fc708106a40 2023-07-20T01:21:46.689 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.755+0000 7fc7057fa700 1 -- 172.21.15.60:0/2296098992 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fc6fc019690 con 0x7fc708106a40 2023-07-20T01:21:46.689 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.755+0000 7fc70e6ed700 1 -- 172.21.15.60:0/2296098992 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc70810a0f0 con 0x7fc708106a40 2023-07-20T01:21:46.690 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.755+0000 7fc7057fa700 1 --2- 172.21.15.60:0/2296098992 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fc6f00387b0 0x7fc6f003ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:46.690 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.755+0000 7fc7057fa700 1 -- 172.21.15.60:0/2296098992 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc6fc04c920 con 0x7fc708106a40 2023-07-20T01:21:46.690 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.755+0000 7fc7077fe700 1 --2- 172.21.15.60:0/2296098992 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fc6f00387b0 0x7fc6f003ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:46.690 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.756+0000 7fc7077fe700 1 --2- 172.21.15.60:0/2296098992 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fc6f00387b0 0x7fc6f003ac70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc6f8006fd0 tx=0x7fc6f8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:46.690 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:45.760+0000 7fc7057fa700 1 -- 172.21.15.60:0/2296098992 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc6fc018db0 con 0x7fc708106a40 2023-07-20T01:21:46.690 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:46.331+0000 7fc70e6ed700 1 -- 172.21.15.60:0/2296098992 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fc70810a310 con 0x7fc708106a40 2023-07-20T01:21:46.690 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:46.333+0000 7fc7057fa700 1 -- 172.21.15.60:0/2296098992 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7fc6fc018350 con 0x7fc708106a40 2023-07-20T01:21:46.690 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:46.336+0000 7fc70e6ed700 1 -- 172.21.15.60:0/2296098992 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fc6f00387b0 msgr2=0x7fc6f003ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:46.691 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:46.336+0000 7fc70e6ed700 1 --2- 172.21.15.60:0/2296098992 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fc6f00387b0 0x7fc6f003ac70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc6f8006fd0 tx=0x7fc6f8006e40 comp rx=0 tx=0).stop 2023-07-20T01:21:46.691 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:46.336+0000 7fc70e6ed700 1 -- 172.21.15.60:0/2296098992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc708106a40 msgr2=0x7fc7081063a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:46.691 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:46.336+0000 7fc70e6ed700 1 --2- 172.21.15.60:0/2296098992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc708106a40 0x7fc7081063a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc006af0 tx=0x7fc6fc011e80 comp rx=0 tx=0).stop 2023-07-20T01:21:46.691 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:46.336+0000 7fc70e6ed700 1 -- 172.21.15.60:0/2296098992 shutdown_connections 2023-07-20T01:21:46.691 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:46.336+0000 7fc70e6ed700 1 --2- 172.21.15.60:0/2296098992 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fc6f00387b0 0x7fc6f003ac70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:46.691 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:46.336+0000 7fc70e6ed700 1 --2- 172.21.15.60:0/2296098992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc708106a40 0x7fc7081063a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:46.691 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:46.336+0000 7fc70e6ed700 1 -- 172.21.15.60:0/2296098992 >> 172.21.15.60:0/2296098992 conn(0x7fc708100410 msgr2=0x7fc708101ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:46.692 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:46.336+0000 7fc70e6ed700 1 -- 172.21.15.60:0/2296098992 shutdown_connections 2023-07-20T01:21:46.692 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2023-07-20T01:21:46.336+0000 7fc70e6ed700 1 -- 172.21.15.60:0/2296098992 wait complete. 2023-07-20T01:21:46.692 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-07-20T01:21:46.692 INFO:teuthology.orchestra.run.smithi060.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-07-20T01:21:46.692 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:21:46.692 INFO:teuthology.orchestra.run.smithi060.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-07-20T01:21:46.692 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:21:46.692 INFO:teuthology.orchestra.run.smithi060.stdout:Or, if you are only running a single cluster on this host: 2023-07-20T01:21:46.693 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:21:46.693 INFO:teuthology.orchestra.run.smithi060.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-07-20T01:21:46.693 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:21:46.693 INFO:teuthology.orchestra.run.smithi060.stdout:Please consider enabling telemetry to help improve Ceph: 2023-07-20T01:21:46.693 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:21:46.693 INFO:teuthology.orchestra.run.smithi060.stdout: ceph telemetry on 2023-07-20T01:21:46.693 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:21:46.693 INFO:teuthology.orchestra.run.smithi060.stdout:For more information see: 2023-07-20T01:21:46.694 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:21:46.694 INFO:teuthology.orchestra.run.smithi060.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2023-07-20T01:21:46.694 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:21:46.694 INFO:teuthology.orchestra.run.smithi060.stdout:Bootstrap complete. 2023-07-20T01:21:46.719 INFO:tasks.cephadm:Fetching config... 2023-07-20T01:21:46.719 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-07-20T01:21:46.720 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-07-20T01:21:46.736 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-07-20T01:21:46.737 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-07-20T01:21:46.737 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-07-20T01:21:46.794 INFO:tasks.cephadm:Fetching mon keyring... 2023-07-20T01:21:46.794 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-07-20T01:21:46.795 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.a/keyring of=/dev/stdout 2023-07-20T01:21:46.863 INFO:tasks.cephadm:Fetching pub ssh key... 2023-07-20T01:21:46.863 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-07-20T01:21:46.863 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-07-20T01:21:46.920 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-07-20T01:21:46.920 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5hIUHf9mSLFFtoqFD3PELkONSJSPyFd15dH1JSzJXBrjCyLmD7+nuw6q+3CxWbmzX5IXq89O79SztAXJkw5TfRRa6jV8sFBLs6Ch0eMSbh30fXWDomymT1L6sDcWMFwKwwE8UHCWN2qsoj6omxciFCUPWYMIHV+xKfc7Hk2DWPNnV31bgE92HjPFK5exGyzrFDYe7fERXdcg1GXylc+7z50WaMYbRTBNGvavOZRPmH5kRVGs3wL1iqpdHuH4IJwkE5UyGAcdghijEXPvTyoVjfXhahnJCrNE/bpeVPmtMBlCFf2HsNzxGAK5ugNCDns2ToOI3FyGc9NFD1ev0vUBH2hgT/Ht9x/9tqKGY33d9/F7PBFmGFB2X0Ky6Vaukp17F+ekERa8B8wuh7Wdcn3ErmLctpMA2CbE5VSm28Rngv9dA9ZEOnESjid82BLCZ/Z34iUGGsCh2TIgK6RwRPLCFmwHj8AcN1qAHQ3UAMM5x5K7NkSUUfvEaqZ4l+UMxWE8= ceph-7daf031e-269b-11ee-9b34-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-20T01:21:47.001 INFO:teuthology.orchestra.run.smithi060.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5hIUHf9mSLFFtoqFD3PELkONSJSPyFd15dH1JSzJXBrjCyLmD7+nuw6q+3CxWbmzX5IXq89O79SztAXJkw5TfRRa6jV8sFBLs6Ch0eMSbh30fXWDomymT1L6sDcWMFwKwwE8UHCWN2qsoj6omxciFCUPWYMIHV+xKfc7Hk2DWPNnV31bgE92HjPFK5exGyzrFDYe7fERXdcg1GXylc+7z50WaMYbRTBNGvavOZRPmH5kRVGs3wL1iqpdHuH4IJwkE5UyGAcdghijEXPvTyoVjfXhahnJCrNE/bpeVPmtMBlCFf2HsNzxGAK5ugNCDns2ToOI3FyGc9NFD1ev0vUBH2hgT/Ht9x/9tqKGY33d9/F7PBFmGFB2X0Ky6Vaukp17F+ekERa8B8wuh7Wdcn3ErmLctpMA2CbE5VSm28Rngv9dA9ZEOnESjid82BLCZ/Z34iUGGsCh2TIgK6RwRPLCFmwHj8AcN1qAHQ3UAMM5x5K7NkSUUfvEaqZ4l+UMxWE8= ceph-7daf031e-269b-11ee-9b34-001a4aab830c 2023-07-20T01:21:47.016 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5hIUHf9mSLFFtoqFD3PELkONSJSPyFd15dH1JSzJXBrjCyLmD7+nuw6q+3CxWbmzX5IXq89O79SztAXJkw5TfRRa6jV8sFBLs6Ch0eMSbh30fXWDomymT1L6sDcWMFwKwwE8UHCWN2qsoj6omxciFCUPWYMIHV+xKfc7Hk2DWPNnV31bgE92HjPFK5exGyzrFDYe7fERXdcg1GXylc+7z50WaMYbRTBNGvavOZRPmH5kRVGs3wL1iqpdHuH4IJwkE5UyGAcdghijEXPvTyoVjfXhahnJCrNE/bpeVPmtMBlCFf2HsNzxGAK5ugNCDns2ToOI3FyGc9NFD1ev0vUBH2hgT/Ht9x/9tqKGY33d9/F7PBFmGFB2X0Ky6Vaukp17F+ekERa8B8wuh7Wdcn3ErmLctpMA2CbE5VSm28Rngv9dA9ZEOnESjid82BLCZ/Z34iUGGsCh2TIgK6RwRPLCFmwHj8AcN1qAHQ3UAMM5x5K7NkSUUfvEaqZ4l+UMxWE8= ceph-7daf031e-269b-11ee-9b34-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-20T01:21:47.061 INFO:teuthology.orchestra.run.smithi136.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5hIUHf9mSLFFtoqFD3PELkONSJSPyFd15dH1JSzJXBrjCyLmD7+nuw6q+3CxWbmzX5IXq89O79SztAXJkw5TfRRa6jV8sFBLs6Ch0eMSbh30fXWDomymT1L6sDcWMFwKwwE8UHCWN2qsoj6omxciFCUPWYMIHV+xKfc7Hk2DWPNnV31bgE92HjPFK5exGyzrFDYe7fERXdcg1GXylc+7z50WaMYbRTBNGvavOZRPmH5kRVGs3wL1iqpdHuH4IJwkE5UyGAcdghijEXPvTyoVjfXhahnJCrNE/bpeVPmtMBlCFf2HsNzxGAK5ugNCDns2ToOI3FyGc9NFD1ev0vUBH2hgT/Ht9x/9tqKGY33d9/F7PBFmGFB2X0Ky6Vaukp17F+ekERa8B8wuh7Wdcn3ErmLctpMA2CbE5VSm28Rngv9dA9ZEOnESjid82BLCZ/Z34iUGGsCh2TIgK6RwRPLCFmwHj8AcN1qAHQ3UAMM5x5K7NkSUUfvEaqZ4l+UMxWE8= ceph-7daf031e-269b-11ee-9b34-001a4aab830c 2023-07-20T01:21:47.074 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5hIUHf9mSLFFtoqFD3PELkONSJSPyFd15dH1JSzJXBrjCyLmD7+nuw6q+3CxWbmzX5IXq89O79SztAXJkw5TfRRa6jV8sFBLs6Ch0eMSbh30fXWDomymT1L6sDcWMFwKwwE8UHCWN2qsoj6omxciFCUPWYMIHV+xKfc7Hk2DWPNnV31bgE92HjPFK5exGyzrFDYe7fERXdcg1GXylc+7z50WaMYbRTBNGvavOZRPmH5kRVGs3wL1iqpdHuH4IJwkE5UyGAcdghijEXPvTyoVjfXhahnJCrNE/bpeVPmtMBlCFf2HsNzxGAK5ugNCDns2ToOI3FyGc9NFD1ev0vUBH2hgT/Ht9x/9tqKGY33d9/F7PBFmGFB2X0Ky6Vaukp17F+ekERa8B8wuh7Wdcn3ErmLctpMA2CbE5VSm28Rngv9dA9ZEOnESjid82BLCZ/Z34iUGGsCh2TIgK6RwRPLCFmwHj8AcN1qAHQ3UAMM5x5K7NkSUUfvEaqZ4l+UMxWE8= ceph-7daf031e-269b-11ee-9b34-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-20T01:21:47.119 INFO:teuthology.orchestra.run.smithi183.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5hIUHf9mSLFFtoqFD3PELkONSJSPyFd15dH1JSzJXBrjCyLmD7+nuw6q+3CxWbmzX5IXq89O79SztAXJkw5TfRRa6jV8sFBLs6Ch0eMSbh30fXWDomymT1L6sDcWMFwKwwE8UHCWN2qsoj6omxciFCUPWYMIHV+xKfc7Hk2DWPNnV31bgE92HjPFK5exGyzrFDYe7fERXdcg1GXylc+7z50WaMYbRTBNGvavOZRPmH5kRVGs3wL1iqpdHuH4IJwkE5UyGAcdghijEXPvTyoVjfXhahnJCrNE/bpeVPmtMBlCFf2HsNzxGAK5ugNCDns2ToOI3FyGc9NFD1ev0vUBH2hgT/Ht9x/9tqKGY33d9/F7PBFmGFB2X0Ky6Vaukp17F+ekERa8B8wuh7Wdcn3ErmLctpMA2CbE5VSm28Rngv9dA9ZEOnESjid82BLCZ/Z34iUGGsCh2TIgK6RwRPLCFmwHj8AcN1qAHQ3UAMM5x5K7NkSUUfvEaqZ4l+UMxWE8= ceph-7daf031e-269b-11ee-9b34-001a4aab830c 2023-07-20T01:21:47.132 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-07-20T01:21:47.603 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:47 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/2296098992' entity='client.admin' 2023-07-20T01:21:47.692 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:21:49.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:49 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:21:49.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:49 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:21:49.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:49 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi060", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:21:49.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:49 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:21:49.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:49 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:21:49.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:49 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:21:49.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:49.861+0000 7f2ff32ff700 1 -- 172.21.15.60:0/2901231442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fec0ff160 msgr2=0x7f2fec0ff580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:49.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:49.861+0000 7f2ff32ff700 1 --2- 172.21.15.60:0/2901231442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fec0ff160 0x7f2fec0ff580 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8009910 tx=0x7f2fe8009c20 comp rx=0 tx=0).stop 2023-07-20T01:21:49.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:49.861+0000 7f2ff32ff700 1 -- 172.21.15.60:0/2901231442 shutdown_connections 2023-07-20T01:21:49.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:49.861+0000 7f2ff32ff700 1 --2- 172.21.15.60:0/2901231442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fec0ff160 0x7f2fec0ff580 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:49.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:49.861+0000 7f2ff32ff700 1 -- 172.21.15.60:0/2901231442 >> 172.21.15.60:0/2901231442 conn(0x7f2fec0fa700 msgr2=0x7f2fec0fcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:49.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:49.861+0000 7f2ff32ff700 1 -- 172.21.15.60:0/2901231442 shutdown_connections 2023-07-20T01:21:49.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:49.862+0000 7f2ff32ff700 1 -- 172.21.15.60:0/2901231442 wait complete. 2023-07-20T01:21:49.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:49.862+0000 7f2ff32ff700 1 Processor -- start 2023-07-20T01:21:49.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:49.862+0000 7f2ff32ff700 1 -- start start 2023-07-20T01:21:49.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:49.862+0000 7f2ff32ff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fec0ff160 0x7f2fec197480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:49.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:49.862+0000 7f2ff32ff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec1979c0 con 0x7f2fec0ff160 2023-07-20T01:21:49.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:49.862+0000 7f2ff109b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fec0ff160 0x7f2fec197480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:49.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:49.862+0000 7f2ff109b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fec0ff160 0x7f2fec197480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42126/0 (socket says 172.21.15.60:42126) 2023-07-20T01:21:49.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:49.862+0000 7f2ff109b700 1 -- 172.21.15.60:0/2637013058 learned_addr learned my addr 172.21.15.60:0/2637013058 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:49.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:49.863+0000 7f2ff109b700 1 -- 172.21.15.60:0/2637013058 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe80095f0 con 0x7f2fec0ff160 2023-07-20T01:21:49.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:49.863+0000 7f2ff109b700 1 --2- 172.21.15.60:0/2637013058 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fec0ff160 0x7f2fec197480 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8005950 tx=0x7f2fe8005810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:49.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:49.863+0000 7f2fe27fc700 1 -- 172.21.15.60:0/2637013058 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2fe800ba90 con 0x7f2fec0ff160 2023-07-20T01:21:49.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:49.863+0000 7f2fe27fc700 1 -- 172.21.15.60:0/2637013058 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f2fe800bbf0 con 0x7f2fec0ff160 2023-07-20T01:21:49.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:49.863+0000 7f2ff32ff700 1 -- 172.21.15.60:0/2637013058 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fec197bc0 con 0x7f2fec0ff160 2023-07-20T01:21:49.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:49.863+0000 7f2fe27fc700 1 -- 172.21.15.60:0/2637013058 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2fe8017440 con 0x7f2fec0ff160 2023-07-20T01:21:49.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:49.863+0000 7f2ff32ff700 1 -- 172.21.15.60:0/2637013058 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fec198060 con 0x7f2fec0ff160 2023-07-20T01:21:49.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:49.864+0000 7f2fe27fc700 1 -- 172.21.15.60:0/2637013058 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f2fe800f460 con 0x7f2fec0ff160 2023-07-20T01:21:49.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:49.864+0000 7f2ff32ff700 1 -- 172.21.15.60:0/2637013058 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fec190f30 con 0x7f2fec0ff160 2023-07-20T01:21:49.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:49.864+0000 7f2fe27fc700 1 --2- 172.21.15.60:0/2637013058 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f2fd8038800 0x7f2fd803acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:49.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:49.864+0000 7f2fe27fc700 1 -- 172.21.15.60:0/2637013058 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2fe804c8a0 con 0x7f2fec0ff160 2023-07-20T01:21:49.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:49.864+0000 7f2ff089a700 1 --2- 172.21.15.60:0/2637013058 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f2fd8038800 0x7f2fd803acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:49.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:49.865+0000 7f2ff089a700 1 --2- 172.21.15.60:0/2637013058 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f2fd8038800 0x7f2fd803acc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc006fd0 tx=0x7f2fdc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:49.850 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:49.869+0000 7f2fe27fc700 1 -- 172.21.15.60:0/2637013058 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2fe802ad30 con 0x7f2fec0ff160 2023-07-20T01:21:50.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:50.069+0000 7f2ff32ff700 1 -- 172.21.15.60:0/2637013058 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f2fec02c970 con 0x7f2fec0ff160 2023-07-20T01:21:50.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:50.072+0000 7f2fe27fc700 1 -- 172.21.15.60:0/2637013058 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f2fe8027030 con 0x7f2fec0ff160 2023-07-20T01:21:50.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:50.074+0000 7f2ff32ff700 1 -- 172.21.15.60:0/2637013058 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f2fd8038800 msgr2=0x7f2fd803acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:50.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:50.074+0000 7f2ff32ff700 1 --2- 172.21.15.60:0/2637013058 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f2fd8038800 0x7f2fd803acc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc006fd0 tx=0x7f2fdc006e40 comp rx=0 tx=0).stop 2023-07-20T01:21:50.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:50.074+0000 7f2ff32ff700 1 -- 172.21.15.60:0/2637013058 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fec0ff160 msgr2=0x7f2fec197480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:50.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:50.074+0000 7f2ff32ff700 1 --2- 172.21.15.60:0/2637013058 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fec0ff160 0x7f2fec197480 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8005950 tx=0x7f2fe8005810 comp rx=0 tx=0).stop 2023-07-20T01:21:50.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:50.074+0000 7f2ff32ff700 1 -- 172.21.15.60:0/2637013058 shutdown_connections 2023-07-20T01:21:50.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:50.074+0000 7f2ff32ff700 1 --2- 172.21.15.60:0/2637013058 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f2fd8038800 0x7f2fd803acc0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:50.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:50.074+0000 7f2ff32ff700 1 --2- 172.21.15.60:0/2637013058 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2fec0ff160 0x7f2fec197480 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:50.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:50.074+0000 7f2ff32ff700 1 -- 172.21.15.60:0/2637013058 >> 172.21.15.60:0/2637013058 conn(0x7f2fec0fa700 msgr2=0x7f2fec068200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:50.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:50.075+0000 7f2ff32ff700 1 -- 172.21.15.60:0/2637013058 shutdown_connections 2023-07-20T01:21:50.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:50.075+0000 7f2ff32ff700 1 -- 172.21.15.60:0/2637013058 wait complete. 2023-07-20T01:21:50.535 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-07-20T01:21:50.535 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-07-20T01:21:51.092 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:21:51.315 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:51 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/2637013058' entity='client.admin' 2023-07-20T01:21:51.315 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:51 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:21:51.315 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:51 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:21:52.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.328+0000 7f12ff992700 1 -- 172.21.15.60:0/1174986594 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12f80ff160 msgr2=0x7f12f80ff580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:52.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.328+0000 7f12ff992700 1 --2- 172.21.15.60:0/1174986594 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12f80ff160 0x7f12f80ff580 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f12e8009910 tx=0x7f12e8009c20 comp rx=0 tx=0).stop 2023-07-20T01:21:52.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.328+0000 7f12ff992700 1 -- 172.21.15.60:0/1174986594 shutdown_connections 2023-07-20T01:21:52.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.328+0000 7f12ff992700 1 --2- 172.21.15.60:0/1174986594 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12f80ff160 0x7f12f80ff580 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:52.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.328+0000 7f12ff992700 1 -- 172.21.15.60:0/1174986594 >> 172.21.15.60:0/1174986594 conn(0x7f12f80fa700 msgr2=0x7f12f80fcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:52.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.328+0000 7f12ff992700 1 -- 172.21.15.60:0/1174986594 shutdown_connections 2023-07-20T01:21:52.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.328+0000 7f12ff992700 1 -- 172.21.15.60:0/1174986594 wait complete. 2023-07-20T01:21:52.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.328+0000 7f12ff992700 1 Processor -- start 2023-07-20T01:21:52.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.329+0000 7f12ff992700 1 -- start start 2023-07-20T01:21:52.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.329+0000 7f12ff992700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12f80ff160 0x7f12f8195280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:52.313 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.329+0000 7f12ff992700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f81957c0 con 0x7f12f80ff160 2023-07-20T01:21:52.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.329+0000 7f12fd72e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12f80ff160 0x7f12f8195280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:52.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.329+0000 7f12fd72e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12f80ff160 0x7f12f8195280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42132/0 (socket says 172.21.15.60:42132) 2023-07-20T01:21:52.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.329+0000 7f12fd72e700 1 -- 172.21.15.60:0/478830368 learned_addr learned my addr 172.21.15.60:0/478830368 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:52.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.329+0000 7f12fd72e700 1 -- 172.21.15.60:0/478830368 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e80095f0 con 0x7f12f80ff160 2023-07-20T01:21:52.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.329+0000 7f12fd72e700 1 --2- 172.21.15.60:0/478830368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12f80ff160 0x7f12f8195280 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f12e8006010 tx=0x7f12e8005810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:52.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.329+0000 7f12ee7fc700 1 -- 172.21.15.60:0/478830368 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f12e800ba90 con 0x7f12f80ff160 2023-07-20T01:21:52.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.329+0000 7f12ee7fc700 1 -- 172.21.15.60:0/478830368 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f12e800bbf0 con 0x7f12f80ff160 2023-07-20T01:21:52.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.329+0000 7f12ff992700 1 -- 172.21.15.60:0/478830368 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12f81959c0 con 0x7f12f80ff160 2023-07-20T01:21:52.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.329+0000 7f12ee7fc700 1 -- 172.21.15.60:0/478830368 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f12e8017440 con 0x7f12f80ff160 2023-07-20T01:21:52.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.330+0000 7f12ff992700 1 -- 172.21.15.60:0/478830368 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f8195e60 con 0x7f12f80ff160 2023-07-20T01:21:52.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.331+0000 7f12ee7fc700 1 -- 172.21.15.60:0/478830368 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f12e800bd60 con 0x7f12f80ff160 2023-07-20T01:21:52.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.331+0000 7f12ff992700 1 -- 172.21.15.60:0/478830368 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12f818eb90 con 0x7f12f80ff160 2023-07-20T01:21:52.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.331+0000 7f12ee7fc700 1 --2- 172.21.15.60:0/478830368 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f12e40387b0 0x7f12e403ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:52.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.331+0000 7f12ee7fc700 1 -- 172.21.15.60:0/478830368 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f12e804c970 con 0x7f12f80ff160 2023-07-20T01:21:52.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.331+0000 7f12fcf2d700 1 --2- 172.21.15.60:0/478830368 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f12e40387b0 0x7f12e403ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:52.316 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.331+0000 7f12fcf2d700 1 --2- 172.21.15.60:0/478830368 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f12e40387b0 0x7f12e403ac70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f12f4006fd0 tx=0x7f12f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:52.318 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.336+0000 7f12ee7fc700 1 -- 172.21.15.60:0/478830368 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f12e802a7d0 con 0x7f12f80ff160 2023-07-20T01:21:52.518 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.536+0000 7f12ff992700 1 -- 172.21.15.60:0/478830368 --> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f12f8060c40 con 0x7f12e40387b0 2023-07-20T01:21:52.521 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.539+0000 7f12ee7fc700 1 -- 172.21.15.60:0/478830368 <== mgr.14152 v2:172.21.15.60:6800/1961833180 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f12f8060c40 con 0x7f12e40387b0 2023-07-20T01:21:52.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.542+0000 7f12ff992700 1 -- 172.21.15.60:0/478830368 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f12e40387b0 msgr2=0x7f12e403ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:52.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.542+0000 7f12ff992700 1 --2- 172.21.15.60:0/478830368 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f12e40387b0 0x7f12e403ac70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f12f4006fd0 tx=0x7f12f4006e40 comp rx=0 tx=0).stop 2023-07-20T01:21:52.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.542+0000 7f12ff992700 1 -- 172.21.15.60:0/478830368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12f80ff160 msgr2=0x7f12f8195280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:52.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.542+0000 7f12ff992700 1 --2- 172.21.15.60:0/478830368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12f80ff160 0x7f12f8195280 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f12e8006010 tx=0x7f12e8005810 comp rx=0 tx=0).stop 2023-07-20T01:21:52.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.542+0000 7f12ff992700 1 -- 172.21.15.60:0/478830368 shutdown_connections 2023-07-20T01:21:52.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.542+0000 7f12ff992700 1 --2- 172.21.15.60:0/478830368 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f12e40387b0 0x7f12e403ac70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:52.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.542+0000 7f12ff992700 1 --2- 172.21.15.60:0/478830368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12f80ff160 0x7f12f8195280 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:52.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.542+0000 7f12ff992700 1 -- 172.21.15.60:0/478830368 >> 172.21.15.60:0/478830368 conn(0x7f12f80fa700 msgr2=0x7f12f8068200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:52.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.542+0000 7f12ff992700 1 -- 172.21.15.60:0/478830368 shutdown_connections 2023-07-20T01:21:52.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:52.542+0000 7f12ff992700 1 -- 172.21.15.60:0/478830368 wait complete. 2023-07-20T01:21:53.055 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi136 2023-07-20T01:21:53.055 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-07-20T01:21:53.055 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/etc/ceph/ceph.conf 2023-07-20T01:21:53.075 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-07-20T01:21:53.076 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T01:21:53.134 INFO:tasks.cephadm:Adding host smithi136 to orchestrator... 2023-07-20T01:21:53.135 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph orch host add smithi136 2023-07-20T01:21:53.704 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:21:53.890 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:53 smithi060 ceph-mon[96524]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:21:53.890 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:53 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:21:53.890 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:53 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:21:53.891 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:53 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:21:53.891 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:53 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:21:53.891 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:53 smithi060 ceph-mon[96524]: Updating smithi060:/etc/ceph/ceph.conf 2023-07-20T01:21:54.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:54 smithi060 ceph-mon[96524]: Updating smithi060:/var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T01:21:54.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:54 smithi060 ceph-mon[96524]: Updating smithi060:/etc/ceph/ceph.client.admin.keyring 2023-07-20T01:21:54.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:54 smithi060 ceph-mon[96524]: Updating smithi060:/var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T01:21:54.930 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:54 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:21:54.930 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:54 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:21:54.930 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:54 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:21:54.984 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:55.002+0000 7f6cf519d700 1 -- 172.21.15.60:0/485394889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6cf0106290 msgr2=0x7f6cf0106670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:54.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:55.002+0000 7f6cf519d700 1 --2- 172.21.15.60:0/485394889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6cf0106290 0x7f6cf0106670 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4009910 tx=0x7f6ce4009c20 comp rx=0 tx=0).stop 2023-07-20T01:21:54.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:55.002+0000 7f6cf519d700 1 -- 172.21.15.60:0/485394889 shutdown_connections 2023-07-20T01:21:54.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:55.002+0000 7f6cf519d700 1 --2- 172.21.15.60:0/485394889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6cf0106290 0x7f6cf0106670 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:54.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:55.002+0000 7f6cf519d700 1 -- 172.21.15.60:0/485394889 >> 172.21.15.60:0/485394889 conn(0x7f6cf0074c70 msgr2=0x7f6cf0075080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:54.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:55.002+0000 7f6cf519d700 1 -- 172.21.15.60:0/485394889 shutdown_connections 2023-07-20T01:21:54.985 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:55.002+0000 7f6cf519d700 1 -- 172.21.15.60:0/485394889 wait complete. 2023-07-20T01:21:54.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:55.003+0000 7f6cf519d700 1 Processor -- start 2023-07-20T01:21:54.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:55.003+0000 7f6cf519d700 1 -- start start 2023-07-20T01:21:54.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:55.003+0000 7f6cf519d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6cf0106290 0x7f6cf01955d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:54.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:55.003+0000 7f6cf519d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf0195b10 con 0x7f6cf0106290 2023-07-20T01:21:54.987 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:55.003+0000 7f6ceed9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6cf0106290 0x7f6cf01955d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:54.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:55.003+0000 7f6ceed9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6cf0106290 0x7f6cf01955d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42138/0 (socket says 172.21.15.60:42138) 2023-07-20T01:21:54.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:55.003+0000 7f6ceed9d700 1 -- 172.21.15.60:0/2111957289 learned_addr learned my addr 172.21.15.60:0/2111957289 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:21:54.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:55.004+0000 7f6ceed9d700 1 -- 172.21.15.60:0/2111957289 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce40095f0 con 0x7f6cf0106290 2023-07-20T01:21:54.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:55.004+0000 7f6ceed9d700 1 --2- 172.21.15.60:0/2111957289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6cf0106290 0x7f6cf01955d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4004f40 tx=0x7f6ce4005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:54.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:55.004+0000 7f6cd7fff700 1 -- 172.21.15.60:0/2111957289 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ce401d070 con 0x7f6cf0106290 2023-07-20T01:21:54.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:55.004+0000 7f6cd7fff700 1 -- 172.21.15.60:0/2111957289 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6ce40044b0 con 0x7f6cf0106290 2023-07-20T01:21:54.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:55.004+0000 7f6cd7fff700 1 -- 172.21.15.60:0/2111957289 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ce4003810 con 0x7f6cf0106290 2023-07-20T01:21:54.988 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:55.004+0000 7f6cf519d700 1 -- 172.21.15.60:0/2111957289 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cf0195d10 con 0x7f6cf0106290 2023-07-20T01:21:54.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:55.004+0000 7f6cf519d700 1 -- 172.21.15.60:0/2111957289 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cf0196190 con 0x7f6cf0106290 2023-07-20T01:21:54.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:55.005+0000 7f6cd7fff700 1 -- 172.21.15.60:0/2111957289 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f6ce4003a40 con 0x7f6cf0106290 2023-07-20T01:21:54.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:55.005+0000 7f6cf519d700 1 -- 172.21.15.60:0/2111957289 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cf0061da0 con 0x7f6cf0106290 2023-07-20T01:21:54.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:55.005+0000 7f6cd7fff700 1 --2- 172.21.15.60:0/2111957289 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f6cd80387f0 0x7f6cd803acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:21:54.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:55.005+0000 7f6cd7fff700 1 -- 172.21.15.60:0/2111957289 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6ce404c7a0 con 0x7f6cf0106290 2023-07-20T01:21:54.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:55.005+0000 7f6cee59c700 1 --2- 172.21.15.60:0/2111957289 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f6cd80387f0 0x7f6cd803acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:21:54.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:55.006+0000 7f6cee59c700 1 --2- 172.21.15.60:0/2111957289 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f6cd80387f0 0x7f6cd803acb0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0006fd0 tx=0x7f6ce0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:21:54.992 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:55.010+0000 7f6cd7fff700 1 -- 172.21.15.60:0/2111957289 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6ce402a770 con 0x7f6cf0106290 2023-07-20T01:21:55.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:55.209+0000 7f6cf519d700 1 -- 172.21.15.60:0/2111957289 --> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi136", "target": ["mon-mgr", ""]}) v1 -- 0x7f6cf00fe470 con 0x7f6cd80387f0 2023-07-20T01:21:56.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:55 smithi060 ceph-mon[96524]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi136", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:21:56.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:55 smithi060 ceph-mon[96524]: Deploying cephadm binary to smithi136 2023-07-20T01:21:57.986 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:58.004+0000 7f6cd7fff700 1 -- 172.21.15.60:0/2111957289 <== mgr.14152 v2:172.21.15.60:6800/1961833180 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f6cf00fe470 con 0x7f6cd80387f0 2023-07-20T01:21:57.987 INFO:teuthology.orchestra.run.smithi060.stdout:Added host 'smithi136' with addr '172.21.15.136' 2023-07-20T01:21:57.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:58.007+0000 7f6cf519d700 1 -- 172.21.15.60:0/2111957289 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f6cd80387f0 msgr2=0x7f6cd803acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:57.989 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:58.007+0000 7f6cf519d700 1 --2- 172.21.15.60:0/2111957289 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f6cd80387f0 0x7f6cd803acb0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0006fd0 tx=0x7f6ce0006e40 comp rx=0 tx=0).stop 2023-07-20T01:21:57.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:58.007+0000 7f6cf519d700 1 -- 172.21.15.60:0/2111957289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6cf0106290 msgr2=0x7f6cf01955d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:21:57.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:58.007+0000 7f6cf519d700 1 --2- 172.21.15.60:0/2111957289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6cf0106290 0x7f6cf01955d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4004f40 tx=0x7f6ce4005020 comp rx=0 tx=0).stop 2023-07-20T01:21:57.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:58.007+0000 7f6cf519d700 1 -- 172.21.15.60:0/2111957289 shutdown_connections 2023-07-20T01:21:57.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:58.007+0000 7f6cf519d700 1 --2- 172.21.15.60:0/2111957289 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f6cd80387f0 0x7f6cd803acb0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:57.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:58.007+0000 7f6cf519d700 1 --2- 172.21.15.60:0/2111957289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6cf0106290 0x7f6cf01955d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:21:57.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:58.007+0000 7f6cf519d700 1 -- 172.21.15.60:0/2111957289 >> 172.21.15.60:0/2111957289 conn(0x7f6cf0074c70 msgr2=0x7f6cf00fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:21:57.990 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:58.007+0000 7f6cf519d700 1 -- 172.21.15.60:0/2111957289 shutdown_connections 2023-07-20T01:21:57.991 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:21:58.007+0000 7f6cf519d700 1 -- 172.21.15.60:0/2111957289 wait complete. 2023-07-20T01:21:58.473 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph orch host ls --format=json 2023-07-20T01:21:59.033 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:21:59.264 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:59 smithi060 ceph-mon[96524]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:21:59.264 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:59 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:21:59.265 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:59 smithi060 ceph-mon[96524]: Added host smithi136 2023-07-20T01:21:59.265 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:59 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:21:59.265 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:21:59 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:00.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.392+0000 7f0c86ce2700 1 -- 172.21.15.60:0/2323950380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0c80105180 msgr2=0x7f0c80105560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:00.376 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.392+0000 7f0c86ce2700 1 --2- 172.21.15.60:0/2323950380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0c80105180 0x7f0c80105560 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0c74009910 tx=0x7f0c74009c20 comp rx=0 tx=0).stop 2023-07-20T01:22:00.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.392+0000 7f0c86ce2700 1 -- 172.21.15.60:0/2323950380 shutdown_connections 2023-07-20T01:22:00.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.392+0000 7f0c86ce2700 1 --2- 172.21.15.60:0/2323950380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0c80105180 0x7f0c80105560 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:00.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.392+0000 7f0c86ce2700 1 -- 172.21.15.60:0/2323950380 >> 172.21.15.60:0/2323950380 conn(0x7f0c80074c70 msgr2=0x7f0c80075080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:22:00.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.392+0000 7f0c86ce2700 1 -- 172.21.15.60:0/2323950380 shutdown_connections 2023-07-20T01:22:00.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.392+0000 7f0c86ce2700 1 -- 172.21.15.60:0/2323950380 wait complete. 2023-07-20T01:22:00.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.393+0000 7f0c86ce2700 1 Processor -- start 2023-07-20T01:22:00.377 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.393+0000 7f0c86ce2700 1 -- start start 2023-07-20T01:22:00.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.393+0000 7f0c86ce2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0c80105180 0x7f0c80195640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:22:00.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.393+0000 7f0c86ce2700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c80195b80 con 0x7f0c80105180 2023-07-20T01:22:00.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.393+0000 7f0c84a7e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0c80105180 0x7f0c80195640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:22:00.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.393+0000 7f0c84a7e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0c80105180 0x7f0c80195640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42164/0 (socket says 172.21.15.60:42164) 2023-07-20T01:22:00.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.393+0000 7f0c84a7e700 1 -- 172.21.15.60:0/668060847 learned_addr learned my addr 172.21.15.60:0/668060847 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:22:00.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.394+0000 7f0c84a7e700 1 -- 172.21.15.60:0/668060847 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c740095f0 con 0x7f0c80105180 2023-07-20T01:22:00.378 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.394+0000 7f0c84a7e700 1 --2- 172.21.15.60:0/668060847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0c80105180 0x7f0c80195640 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0c74006010 tx=0x7f0c74004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:22:00.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.394+0000 7f0c7dffb700 1 -- 172.21.15.60:0/668060847 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0c7401d070 con 0x7f0c80105180 2023-07-20T01:22:00.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.394+0000 7f0c7dffb700 1 -- 172.21.15.60:0/668060847 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0c7400bdf0 con 0x7f0c80105180 2023-07-20T01:22:00.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.394+0000 7f0c86ce2700 1 -- 172.21.15.60:0/668060847 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c80195de0 con 0x7f0c80105180 2023-07-20T01:22:00.379 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.394+0000 7f0c7dffb700 1 -- 172.21.15.60:0/668060847 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0c74022d80 con 0x7f0c80105180 2023-07-20T01:22:00.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.394+0000 7f0c86ce2700 1 -- 172.21.15.60:0/668060847 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c80196260 con 0x7f0c80105180 2023-07-20T01:22:00.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.395+0000 7f0c7dffb700 1 -- 172.21.15.60:0/668060847 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f0c7400f460 con 0x7f0c80105180 2023-07-20T01:22:00.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.395+0000 7f0c7dffb700 1 --2- 172.21.15.60:0/668060847 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f0c70038800 0x7f0c7003acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:22:00.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.395+0000 7f0c7dffb700 1 -- 172.21.15.60:0/668060847 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0c7404c840 con 0x7f0c80105180 2023-07-20T01:22:00.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.395+0000 7f0c7ffff700 1 --2- 172.21.15.60:0/668060847 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f0c70038800 0x7f0c7003acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:22:00.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.395+0000 7f0c86ce2700 1 -- 172.21.15.60:0/668060847 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c80061da0 con 0x7f0c80105180 2023-07-20T01:22:00.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.396+0000 7f0c7ffff700 1 --2- 172.21.15.60:0/668060847 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f0c70038800 0x7f0c7003acc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c006fd0 tx=0x7f0c6c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:22:00.382 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.400+0000 7f0c7dffb700 1 -- 172.21.15.60:0/668060847 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0c74027030 con 0x7f0c80105180 2023-07-20T01:22:00.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:00 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:00.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.617+0000 7f0c86ce2700 1 -- 172.21.15.60:0/668060847 --> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0c80104450 con 0x7f0c70038800 2023-07-20T01:22:00.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.618+0000 7f0c7dffb700 1 -- 172.21.15.60:0/668060847 <== mgr.14152 v2:172.21.15.60:6800/1961833180 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f0c80104450 con 0x7f0c70038800 2023-07-20T01:22:00.600 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:22:00.601 INFO:teuthology.orchestra.run.smithi060.stdout:[{"addr": "172.21.15.60", "hostname": "smithi060", "labels": [], "status": ""}, {"addr": "172.21.15.136", "hostname": "smithi136", "labels": [], "status": ""}] 2023-07-20T01:22:00.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.620+0000 7f0c86ce2700 1 -- 172.21.15.60:0/668060847 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f0c70038800 msgr2=0x7f0c7003acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:00.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.620+0000 7f0c86ce2700 1 --2- 172.21.15.60:0/668060847 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f0c70038800 0x7f0c7003acc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c006fd0 tx=0x7f0c6c006e40 comp rx=0 tx=0).stop 2023-07-20T01:22:00.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.620+0000 7f0c86ce2700 1 -- 172.21.15.60:0/668060847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0c80105180 msgr2=0x7f0c80195640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:00.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.620+0000 7f0c86ce2700 1 --2- 172.21.15.60:0/668060847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0c80105180 0x7f0c80195640 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0c74006010 tx=0x7f0c74004ff0 comp rx=0 tx=0).stop 2023-07-20T01:22:00.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.620+0000 7f0c86ce2700 1 -- 172.21.15.60:0/668060847 shutdown_connections 2023-07-20T01:22:00.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.620+0000 7f0c86ce2700 1 --2- 172.21.15.60:0/668060847 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f0c70038800 0x7f0c7003acc0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:00.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.620+0000 7f0c86ce2700 1 --2- 172.21.15.60:0/668060847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0c80105180 0x7f0c80195640 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:00.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.620+0000 7f0c86ce2700 1 -- 172.21.15.60:0/668060847 >> 172.21.15.60:0/668060847 conn(0x7f0c80074c70 msgr2=0x7f0c80102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:22:00.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.620+0000 7f0c86ce2700 1 -- 172.21.15.60:0/668060847 shutdown_connections 2023-07-20T01:22:00.604 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:00.620+0000 7f0c86ce2700 1 -- 172.21.15.60:0/668060847 wait complete. 2023-07-20T01:22:00.998 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi183 2023-07-20T01:22:00.998 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2023-07-20T01:22:00.998 DEBUG:teuthology.orchestra.run.smithi183:> dd of=/etc/ceph/ceph.conf 2023-07-20T01:22:01.019 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2023-07-20T01:22:01.020 DEBUG:teuthology.orchestra.run.smithi183:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T01:22:01.078 INFO:tasks.cephadm:Adding host smithi183 to orchestrator... 2023-07-20T01:22:01.078 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph orch host add smithi183 2023-07-20T01:22:01.639 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:22:01.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:01 smithi060 ceph-mon[96524]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:01.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:01 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:01.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:01 smithi060 ceph-mon[96524]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T01:22:02.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:02.849+0000 7fe8b4276700 1 -- 172.21.15.60:0/2898824546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8ac104120 msgr2=0x7fe8ac104500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:02.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:02.849+0000 7fe8b4276700 1 --2- 172.21.15.60:0/2898824546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8ac104120 0x7fe8ac104500 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe8a8009910 tx=0x7fe8a8009c20 comp rx=0 tx=0).stop 2023-07-20T01:22:02.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:02.849+0000 7fe8b4276700 1 -- 172.21.15.60:0/2898824546 shutdown_connections 2023-07-20T01:22:02.832 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:02.849+0000 7fe8b4276700 1 --2- 172.21.15.60:0/2898824546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8ac104120 0x7fe8ac104500 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:02.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:02.849+0000 7fe8b4276700 1 -- 172.21.15.60:0/2898824546 >> 172.21.15.60:0/2898824546 conn(0x7fe8ac0757d0 msgr2=0x7fe8ac075be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:22:02.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:02.849+0000 7fe8b4276700 1 -- 172.21.15.60:0/2898824546 shutdown_connections 2023-07-20T01:22:02.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:02.849+0000 7fe8b4276700 1 -- 172.21.15.60:0/2898824546 wait complete. 2023-07-20T01:22:02.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:02.849+0000 7fe8b4276700 1 Processor -- start 2023-07-20T01:22:02.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:02.849+0000 7fe8b4276700 1 -- start start 2023-07-20T01:22:02.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:02.850+0000 7fe8b4276700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8ac104120 0x7fe8ac196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:22:02.833 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:02.850+0000 7fe8b4276700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8ac196970 con 0x7fe8ac104120 2023-07-20T01:22:02.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:02.850+0000 7fe8b2012700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8ac104120 0x7fe8ac196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:22:02.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:02.850+0000 7fe8b2012700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8ac104120 0x7fe8ac196430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42170/0 (socket says 172.21.15.60:42170) 2023-07-20T01:22:02.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:02.850+0000 7fe8b2012700 1 -- 172.21.15.60:0/2243647184 learned_addr learned my addr 172.21.15.60:0/2243647184 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:22:02.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:02.850+0000 7fe8b2012700 1 -- 172.21.15.60:0/2243647184 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a80095f0 con 0x7fe8ac104120 2023-07-20T01:22:02.834 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:02.850+0000 7fe8b2012700 1 --2- 172.21.15.60:0/2243647184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8ac104120 0x7fe8ac196430 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe8a8004f40 tx=0x7fe8a8005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:22:02.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:02.850+0000 7fe8a37fe700 1 -- 172.21.15.60:0/2243647184 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe8a801d070 con 0x7fe8ac104120 2023-07-20T01:22:02.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:02.850+0000 7fe8a37fe700 1 -- 172.21.15.60:0/2243647184 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe8a80044b0 con 0x7fe8ac104120 2023-07-20T01:22:02.835 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:02.850+0000 7fe8b4276700 1 -- 172.21.15.60:0/2243647184 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8ac196b70 con 0x7fe8ac104120 2023-07-20T01:22:02.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:02.850+0000 7fe8a37fe700 1 -- 172.21.15.60:0/2243647184 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe8a8003860 con 0x7fe8ac104120 2023-07-20T01:22:02.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:02.850+0000 7fe8b4276700 1 -- 172.21.15.60:0/2243647184 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8ac196fb0 con 0x7fe8ac104120 2023-07-20T01:22:02.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:02.851+0000 7fe8b4276700 1 -- 172.21.15.60:0/2243647184 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8ac103500 con 0x7fe8ac104120 2023-07-20T01:22:02.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:02.851+0000 7fe8a37fe700 1 -- 172.21.15.60:0/2243647184 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fe8a8003ae0 con 0x7fe8ac104120 2023-07-20T01:22:02.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:02.852+0000 7fe8a37fe700 1 --2- 172.21.15.60:0/2243647184 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fe8980387b0 0x7fe89803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:22:02.836 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:02.852+0000 7fe8a37fe700 1 -- 172.21.15.60:0/2243647184 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe8a804c680 con 0x7fe8ac104120 2023-07-20T01:22:02.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:02.852+0000 7fe8b1811700 1 --2- 172.21.15.60:0/2243647184 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fe8980387b0 0x7fe89803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:22:02.837 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:02.852+0000 7fe8b1811700 1 --2- 172.21.15.60:0/2243647184 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fe8980387b0 0x7fe89803ac70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe89c006fd0 tx=0x7fe89c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:22:02.838 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:02.856+0000 7fe8a37fe700 1 -- 172.21.15.60:0/2243647184 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe8a802a540 con 0x7fe8ac104120 2023-07-20T01:22:03.045 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:03.063+0000 7fe8b4276700 1 -- 172.21.15.60:0/2243647184 --> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi183", "target": ["mon-mgr", ""]}) v1 -- 0x7fe8ac0fed60 con 0x7fe8980387b0 2023-07-20T01:22:03.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:03 smithi060 ceph-mon[96524]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:03.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:03 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:03.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:03 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:03.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:03 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:03.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:03 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:03.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:03 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi136", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:22:03.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:03 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:03.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:03 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:22:04.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:04 smithi060 ceph-mon[96524]: Updating smithi136:/etc/ceph/ceph.conf 2023-07-20T01:22:04.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:04 smithi060 ceph-mon[96524]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi183", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:22:04.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:04 smithi060 ceph-mon[96524]: Updating smithi136:/var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T01:22:04.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:04 smithi060 ceph-mon[96524]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2023-07-20T01:22:04.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:04 smithi060 ceph-mon[96524]: Deploying cephadm binary to smithi183 2023-07-20T01:22:04.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:04 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:04.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:04 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:04.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:04 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:05.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:05 smithi060 ceph-mon[96524]: Updating smithi136:/var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T01:22:05.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:05 smithi060 ceph-mon[96524]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:05.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:05.859+0000 7fe8a37fe700 1 -- 172.21.15.60:0/2243647184 <== mgr.14152 v2:172.21.15.60:6800/1961833180 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fe8ac0fed60 con 0x7fe8980387b0 2023-07-20T01:22:05.841 INFO:teuthology.orchestra.run.smithi060.stdout:Added host 'smithi183' with addr '172.21.15.183' 2023-07-20T01:22:05.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:05.861+0000 7fe8b4276700 1 -- 172.21.15.60:0/2243647184 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fe8980387b0 msgr2=0x7fe89803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:05.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:05.861+0000 7fe8b4276700 1 --2- 172.21.15.60:0/2243647184 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fe8980387b0 0x7fe89803ac70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe89c006fd0 tx=0x7fe89c006e40 comp rx=0 tx=0).stop 2023-07-20T01:22:05.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:05.861+0000 7fe8b4276700 1 -- 172.21.15.60:0/2243647184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8ac104120 msgr2=0x7fe8ac196430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:05.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:05.861+0000 7fe8b4276700 1 --2- 172.21.15.60:0/2243647184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8ac104120 0x7fe8ac196430 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe8a8004f40 tx=0x7fe8a8005020 comp rx=0 tx=0).stop 2023-07-20T01:22:05.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:05.861+0000 7fe8b4276700 1 -- 172.21.15.60:0/2243647184 shutdown_connections 2023-07-20T01:22:05.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:05.861+0000 7fe8b4276700 1 --2- 172.21.15.60:0/2243647184 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fe8980387b0 0x7fe89803ac70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:05.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:05.861+0000 7fe8b4276700 1 --2- 172.21.15.60:0/2243647184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8ac104120 0x7fe8ac196430 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:05.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:05.861+0000 7fe8b4276700 1 -- 172.21.15.60:0/2243647184 >> 172.21.15.60:0/2243647184 conn(0x7fe8ac0757d0 msgr2=0x7fe8ac0fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:22:05.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:05.861+0000 7fe8b4276700 1 -- 172.21.15.60:0/2243647184 shutdown_connections 2023-07-20T01:22:05.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:05.861+0000 7fe8b4276700 1 -- 172.21.15.60:0/2243647184 wait complete. 2023-07-20T01:22:06.306 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph orch host ls --format=json 2023-07-20T01:22:06.864 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:22:07.111 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:06 smithi060 ceph-mon[96524]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:07.112 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:06 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:07.112 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:06 smithi060 ceph-mon[96524]: Added host smithi183 2023-07-20T01:22:07.112 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:06 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:22:07.112 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:06 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:08.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.073+0000 7f745dcdc700 1 -- 172.21.15.60:0/1957599204 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74580ff340 msgr2=0x7f74580ff720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:08.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.073+0000 7f745dcdc700 1 --2- 172.21.15.60:0/1957599204 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74580ff340 0x7f74580ff720 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7448009910 tx=0x7f7448009c20 comp rx=0 tx=0).stop 2023-07-20T01:22:08.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.074+0000 7f745dcdc700 1 -- 172.21.15.60:0/1957599204 shutdown_connections 2023-07-20T01:22:08.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.074+0000 7f745dcdc700 1 --2- 172.21.15.60:0/1957599204 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74580ff340 0x7f74580ff720 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:08.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.074+0000 7f745dcdc700 1 -- 172.21.15.60:0/1957599204 >> 172.21.15.60:0/1957599204 conn(0x7f74580fabf0 msgr2=0x7f74580fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:22:08.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.074+0000 7f745dcdc700 1 -- 172.21.15.60:0/1957599204 shutdown_connections 2023-07-20T01:22:08.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.074+0000 7f745dcdc700 1 -- 172.21.15.60:0/1957599204 wait complete. 2023-07-20T01:22:08.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.074+0000 7f745dcdc700 1 Processor -- start 2023-07-20T01:22:08.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.074+0000 7f745dcdc700 1 -- start start 2023-07-20T01:22:08.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.075+0000 7f745dcdc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74580ff340 0x7f7458195620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:22:08.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.075+0000 7f745dcdc700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7458195b60 con 0x7f74580ff340 2023-07-20T01:22:08.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.075+0000 7f74577fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74580ff340 0x7f7458195620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:22:08.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.075+0000 7f74577fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74580ff340 0x7f7458195620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42196/0 (socket says 172.21.15.60:42196) 2023-07-20T01:22:08.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.075+0000 7f74577fe700 1 -- 172.21.15.60:0/1623390132 learned_addr learned my addr 172.21.15.60:0/1623390132 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:22:08.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.075+0000 7f74577fe700 1 -- 172.21.15.60:0/1623390132 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74480095f0 con 0x7f74580ff340 2023-07-20T01:22:08.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.075+0000 7f74577fe700 1 --2- 172.21.15.60:0/1623390132 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74580ff340 0x7f7458195620 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7448006010 tx=0x7f7448004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:22:08.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.075+0000 7f7454ff9700 1 -- 172.21.15.60:0/1623390132 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f744801d070 con 0x7f74580ff340 2023-07-20T01:22:08.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.075+0000 7f7454ff9700 1 -- 172.21.15.60:0/1623390132 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f744800bdf0 con 0x7f74580ff340 2023-07-20T01:22:08.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.075+0000 7f7454ff9700 1 -- 172.21.15.60:0/1623390132 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7448022d80 con 0x7f74580ff340 2023-07-20T01:22:08.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.075+0000 7f745dcdc700 1 -- 172.21.15.60:0/1623390132 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7458195d60 con 0x7f74580ff340 2023-07-20T01:22:08.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.075+0000 7f745dcdc700 1 -- 172.21.15.60:0/1623390132 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7458196180 con 0x7f74580ff340 2023-07-20T01:22:08.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.076+0000 7f7454ff9700 1 -- 172.21.15.60:0/1623390132 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f744800b960 con 0x7f74580ff340 2023-07-20T01:22:08.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.076+0000 7f745dcdc700 1 -- 172.21.15.60:0/1623390132 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7458061da0 con 0x7f74580ff340 2023-07-20T01:22:08.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.077+0000 7f7454ff9700 1 --2- 172.21.15.60:0/1623390132 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f7440038800 0x7f744003acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:22:08.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.077+0000 7f7454ff9700 1 -- 172.21.15.60:0/1623390132 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f744804c830 con 0x7f74580ff340 2023-07-20T01:22:08.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.077+0000 7f7456ffd700 1 --2- 172.21.15.60:0/1623390132 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f7440038800 0x7f744003acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:22:08.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.077+0000 7f7456ffd700 1 --2- 172.21.15.60:0/1623390132 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f7440038800 0x7f744003acc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f744c006fd0 tx=0x7f744c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:22:08.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.082+0000 7f7454ff9700 1 -- 172.21.15.60:0/1623390132 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f744802b590 con 0x7f74580ff340 2023-07-20T01:22:08.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:07 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:08.283 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.301+0000 7f745dcdc700 1 -- 172.21.15.60:0/1623390132 --> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7458068910 con 0x7f7440038800 2023-07-20T01:22:08.284 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.303+0000 7f7454ff9700 1 -- 172.21.15.60:0/1623390132 <== mgr.14152 v2:172.21.15.60:6800/1961833180 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+240 (secure 0 0 0) 0x7f7458068910 con 0x7f7440038800 2023-07-20T01:22:08.285 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:22:08.285 INFO:teuthology.orchestra.run.smithi060.stdout:[{"addr": "172.21.15.60", "hostname": "smithi060", "labels": [], "status": ""}, {"addr": "172.21.15.136", "hostname": "smithi136", "labels": [], "status": ""}, {"addr": "172.21.15.183", "hostname": "smithi183", "labels": [], "status": ""}] 2023-07-20T01:22:08.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.305+0000 7f745dcdc700 1 -- 172.21.15.60:0/1623390132 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f7440038800 msgr2=0x7f744003acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:08.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.305+0000 7f745dcdc700 1 --2- 172.21.15.60:0/1623390132 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f7440038800 0x7f744003acc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f744c006fd0 tx=0x7f744c006e40 comp rx=0 tx=0).stop 2023-07-20T01:22:08.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.305+0000 7f745dcdc700 1 -- 172.21.15.60:0/1623390132 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74580ff340 msgr2=0x7f7458195620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:08.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.305+0000 7f745dcdc700 1 --2- 172.21.15.60:0/1623390132 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74580ff340 0x7f7458195620 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7448006010 tx=0x7f7448004ff0 comp rx=0 tx=0).stop 2023-07-20T01:22:08.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.305+0000 7f745dcdc700 1 -- 172.21.15.60:0/1623390132 shutdown_connections 2023-07-20T01:22:08.287 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.305+0000 7f745dcdc700 1 --2- 172.21.15.60:0/1623390132 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f7440038800 0x7f744003acc0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:08.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.305+0000 7f745dcdc700 1 --2- 172.21.15.60:0/1623390132 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74580ff340 0x7f7458195620 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:08.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.305+0000 7f745dcdc700 1 -- 172.21.15.60:0/1623390132 >> 172.21.15.60:0/1623390132 conn(0x7f74580fabf0 msgr2=0x7f7458068200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:22:08.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.305+0000 7f745dcdc700 1 -- 172.21.15.60:0/1623390132 shutdown_connections 2023-07-20T01:22:08.288 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:08.305+0000 7f745dcdc700 1 -- 172.21.15.60:0/1623390132 wait complete. 2023-07-20T01:22:08.728 INFO:tasks.cephadm:Setting crush tunables to default 2023-07-20T01:22:08.728 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph osd crush tunables default 2023-07-20T01:22:09.290 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:22:09.512 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:09 smithi060 ceph-mon[96524]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:09.513 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:09 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:09.513 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:09 smithi060 ceph-mon[96524]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T01:22:10.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:10.609+0000 7f8d497f7700 1 -- 172.21.15.60:0/3385913063 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d440fe960 msgr2=0x7f8d440fed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:10.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:10.609+0000 7f8d497f7700 1 --2- 172.21.15.60:0/3385913063 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d440fe960 0x7f8d440fed40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8d34009910 tx=0x7f8d34009c20 comp rx=0 tx=0).stop 2023-07-20T01:22:10.592 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:10.609+0000 7f8d497f7700 1 -- 172.21.15.60:0/3385913063 shutdown_connections 2023-07-20T01:22:10.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:10.609+0000 7f8d497f7700 1 --2- 172.21.15.60:0/3385913063 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d440fe960 0x7f8d440fed40 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:10.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:10.609+0000 7f8d497f7700 1 -- 172.21.15.60:0/3385913063 >> 172.21.15.60:0/3385913063 conn(0x7f8d440757d0 msgr2=0x7f8d44075be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:22:10.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:10.609+0000 7f8d497f7700 1 -- 172.21.15.60:0/3385913063 shutdown_connections 2023-07-20T01:22:10.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:10.609+0000 7f8d497f7700 1 -- 172.21.15.60:0/3385913063 wait complete. 2023-07-20T01:22:10.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:10.610+0000 7f8d497f7700 1 Processor -- start 2023-07-20T01:22:10.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:10.610+0000 7f8d497f7700 1 -- start start 2023-07-20T01:22:10.593 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:10.610+0000 7f8d497f7700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d440fe960 0x7f8d441955f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:22:10.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:10.610+0000 7f8d497f7700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d44195b30 con 0x7f8d440fe960 2023-07-20T01:22:10.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:10.610+0000 7f8d42ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d440fe960 0x7f8d441955f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:22:10.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:10.610+0000 7f8d42ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d440fe960 0x7f8d441955f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42202/0 (socket says 172.21.15.60:42202) 2023-07-20T01:22:10.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:10.610+0000 7f8d42ffd700 1 -- 172.21.15.60:0/1820302194 learned_addr learned my addr 172.21.15.60:0/1820302194 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:22:10.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:10.610+0000 7f8d42ffd700 1 -- 172.21.15.60:0/1820302194 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d340095f0 con 0x7f8d440fe960 2023-07-20T01:22:10.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:10.611+0000 7f8d42ffd700 1 --2- 172.21.15.60:0/1820302194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d440fe960 0x7f8d441955f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8d34006010 tx=0x7f8d34004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:22:10.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:10.611+0000 7f8d2bfff700 1 -- 172.21.15.60:0/1820302194 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8d3401d070 con 0x7f8d440fe960 2023-07-20T01:22:10.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:10.611+0000 7f8d2bfff700 1 -- 172.21.15.60:0/1820302194 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8d3400bdf0 con 0x7f8d440fe960 2023-07-20T01:22:10.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:10.611+0000 7f8d497f7700 1 -- 172.21.15.60:0/1820302194 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d44195d30 con 0x7f8d440fe960 2023-07-20T01:22:10.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:10.611+0000 7f8d2bfff700 1 -- 172.21.15.60:0/1820302194 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8d34022d80 con 0x7f8d440fe960 2023-07-20T01:22:10.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:10.611+0000 7f8d497f7700 1 -- 172.21.15.60:0/1820302194 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d44196150 con 0x7f8d440fe960 2023-07-20T01:22:10.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:10.612+0000 7f8d2bfff700 1 -- 172.21.15.60:0/1820302194 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f8d3400f460 con 0x7f8d440fe960 2023-07-20T01:22:10.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:10.612+0000 7f8d497f7700 1 -- 172.21.15.60:0/1820302194 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d44061da0 con 0x7f8d440fe960 2023-07-20T01:22:10.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:10.612+0000 7f8d2bfff700 1 --2- 172.21.15.60:0/1820302194 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f8d2c0387b0 0x7f8d2c03ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:22:10.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:10.612+0000 7f8d2bfff700 1 -- 172.21.15.60:0/1820302194 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8d3404c9b0 con 0x7f8d440fe960 2023-07-20T01:22:10.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:10.612+0000 7f8d427fc700 1 --2- 172.21.15.60:0/1820302194 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f8d2c0387b0 0x7f8d2c03ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:22:10.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:10.612+0000 7f8d427fc700 1 --2- 172.21.15.60:0/1820302194 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f8d2c0387b0 0x7f8d2c03ac70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8d38006fd0 tx=0x7f8d38006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:22:10.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:10.617+0000 7f8d2bfff700 1 -- 172.21.15.60:0/1820302194 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8d34020960 con 0x7f8d440fe960 2023-07-20T01:22:10.803 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:10.821+0000 7f8d497f7700 1 -- 172.21.15.60:0/1820302194 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f8d44196950 con 0x7f8d440fe960 2023-07-20T01:22:11.199 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:11.217+0000 7f8d2bfff700 1 -- 172.21.15.60:0/1820302194 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f8d34027030 con 0x7f8d440fe960 2023-07-20T01:22:11.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:11.220+0000 7f8d497f7700 1 -- 172.21.15.60:0/1820302194 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f8d2c0387b0 msgr2=0x7f8d2c03ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:11.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:11.220+0000 7f8d497f7700 1 --2- 172.21.15.60:0/1820302194 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f8d2c0387b0 0x7f8d2c03ac70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8d38006fd0 tx=0x7f8d38006e40 comp rx=0 tx=0).stop 2023-07-20T01:22:11.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:11.220+0000 7f8d497f7700 1 -- 172.21.15.60:0/1820302194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d440fe960 msgr2=0x7f8d441955f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:11.202 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:11.220+0000 7f8d497f7700 1 --2- 172.21.15.60:0/1820302194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d440fe960 0x7f8d441955f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8d34006010 tx=0x7f8d34004ff0 comp rx=0 tx=0).stop 2023-07-20T01:22:11.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:11.220+0000 7f8d497f7700 1 -- 172.21.15.60:0/1820302194 shutdown_connections 2023-07-20T01:22:11.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:11.220+0000 7f8d497f7700 1 --2- 172.21.15.60:0/1820302194 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f8d2c0387b0 0x7f8d2c03ac70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:11.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:11.220+0000 7f8d497f7700 1 --2- 172.21.15.60:0/1820302194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d440fe960 0x7f8d441955f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:11.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:11.220+0000 7f8d497f7700 1 -- 172.21.15.60:0/1820302194 >> 172.21.15.60:0/1820302194 conn(0x7f8d440757d0 msgr2=0x7f8d441046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:22:11.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:11.220+0000 7f8d497f7700 1 -- 172.21.15.60:0/1820302194 shutdown_connections 2023-07-20T01:22:11.203 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:11.220+0000 7f8d497f7700 1 -- 172.21.15.60:0/1820302194 wait complete. 2023-07-20T01:22:11.204 INFO:teuthology.orchestra.run.smithi060.stderr:adjusted tunables profile to default 2023-07-20T01:22:11.539 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:11 smithi060 ceph-mon[96524]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:11.539 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:11 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/1820302194' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-20T01:22:11.540 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:11 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:11.540 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:11 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:11.540 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:11 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:11.541 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:11 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:11.541 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:11 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:22:11.541 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:11 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:11.541 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:11 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:22:11.637 INFO:tasks.cephadm:Adding mon.a on smithi060 2023-07-20T01:22:11.637 INFO:tasks.cephadm:Adding mon.b on smithi136 2023-07-20T01:22:11.638 INFO:tasks.cephadm:Adding mon.c on smithi183 2023-07-20T01:22:11.638 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph orch apply mon '3;smithi060:172.21.15.60=a;smithi136:172.21.15.136=b;smithi183:172.21.15.183=c' 2023-07-20T01:22:12.218 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T01:22:12.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:12 smithi060 ceph-mon[96524]: Updating smithi183:/etc/ceph/ceph.conf 2023-07-20T01:22:12.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:12 smithi060 ceph-mon[96524]: Updating smithi183:/var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T01:22:12.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:12 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/1820302194' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-20T01:22:12.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:12 smithi060 ceph-mon[96524]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T01:22:12.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:12 smithi060 ceph-mon[96524]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2023-07-20T01:22:12.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:12 smithi060 ceph-mon[96524]: Updating smithi183:/var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T01:22:12.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:12 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:12.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:12 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:12.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:12 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:13.475 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.492+0000 7f5ff5675700 1 -- 172.21.15.183:0/4000225737 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ff0105180 msgr2=0x7f5ff0105560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:13.476 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.492+0000 7f5ff5675700 1 --2- 172.21.15.183:0/4000225737 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ff0105180 0x7f5ff0105560 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5fe0009910 tx=0x7f5fe0009c20 comp rx=0 tx=0).stop 2023-07-20T01:22:13.476 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.493+0000 7f5ff5675700 1 -- 172.21.15.183:0/4000225737 shutdown_connections 2023-07-20T01:22:13.476 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.493+0000 7f5ff5675700 1 --2- 172.21.15.183:0/4000225737 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ff0105180 0x7f5ff0105560 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:13.476 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.493+0000 7f5ff5675700 1 -- 172.21.15.183:0/4000225737 >> 172.21.15.183:0/4000225737 conn(0x7f5ff0074c70 msgr2=0x7f5ff0075080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:22:13.476 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.493+0000 7f5ff5675700 1 -- 172.21.15.183:0/4000225737 shutdown_connections 2023-07-20T01:22:13.476 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.493+0000 7f5ff5675700 1 -- 172.21.15.183:0/4000225737 wait complete. 2023-07-20T01:22:13.477 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.493+0000 7f5ff5675700 1 Processor -- start 2023-07-20T01:22:13.477 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.493+0000 7f5ff5675700 1 -- start start 2023-07-20T01:22:13.477 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.494+0000 7f5ff5675700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ff0105180 0x7f5ff0195640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:22:13.477 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.494+0000 7f5ff5675700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff0195b80 con 0x7f5ff0105180 2023-07-20T01:22:13.477 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.494+0000 7f5feeffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ff0105180 0x7f5ff0195640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:22:13.477 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.494+0000 7f5feeffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ff0105180 0x7f5ff0195640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.183:47640/0 (socket says 172.21.15.183:47640) 2023-07-20T01:22:13.478 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.494+0000 7f5feeffd700 1 -- 172.21.15.183:0/1174868351 learned_addr learned my addr 172.21.15.183:0/1174868351 (peer_addr_for_me v2:172.21.15.183:0/0) 2023-07-20T01:22:13.479 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.494+0000 7f5feeffd700 1 -- 172.21.15.183:0/1174868351 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe00095f0 con 0x7f5ff0105180 2023-07-20T01:22:13.479 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.494+0000 7f5feeffd700 1 --2- 172.21.15.183:0/1174868351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ff0105180 0x7f5ff0195640 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5fe0004f40 tx=0x7f5fe0005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:22:13.479 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.494+0000 7f5fe7fff700 1 -- 172.21.15.183:0/1174868351 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5fe001d070 con 0x7f5ff0105180 2023-07-20T01:22:13.479 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.494+0000 7f5ff5675700 1 -- 172.21.15.183:0/1174868351 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ff0195d80 con 0x7f5ff0105180 2023-07-20T01:22:13.479 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.495+0000 7f5fe7fff700 1 -- 172.21.15.183:0/1174868351 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5fe00044b0 con 0x7f5ff0105180 2023-07-20T01:22:13.479 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.495+0000 7f5fe7fff700 1 -- 172.21.15.183:0/1174868351 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5fe0003810 con 0x7f5ff0105180 2023-07-20T01:22:13.480 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.495+0000 7f5ff5675700 1 -- 172.21.15.183:0/1174868351 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ff0196200 con 0x7f5ff0105180 2023-07-20T01:22:13.480 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.495+0000 7f5ff5675700 1 -- 172.21.15.183:0/1174868351 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ff0061da0 con 0x7f5ff0105180 2023-07-20T01:22:13.480 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.495+0000 7f5fe7fff700 1 -- 172.21.15.183:0/1174868351 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f5fe000b960 con 0x7f5ff0105180 2023-07-20T01:22:13.480 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.496+0000 7f5fe7fff700 1 --2- 172.21.15.183:0/1174868351 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f5fdc0387a0 0x7f5fdc03ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:22:13.480 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.496+0000 7f5fe7fff700 1 -- 172.21.15.183:0/1174868351 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5fe004c940 con 0x7f5ff0105180 2023-07-20T01:22:13.480 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.496+0000 7f5fee7fc700 1 --2- 172.21.15.183:0/1174868351 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f5fdc0387a0 0x7f5fdc03ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:22:13.480 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.496+0000 7f5fee7fc700 1 --2- 172.21.15.183:0/1174868351 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f5fdc0387a0 0x7f5fdc03ac60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5fd8006fd0 tx=0x7f5fd8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:22:13.482 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.501+0000 7f5fe7fff700 1 -- 172.21.15.183:0/1174868351 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5fe002a540 con 0x7f5ff0105180 2023-07-20T01:22:13.674 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.693+0000 7f5ff5675700 1 -- 172.21.15.183:0/1174868351 --> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi060:172.21.15.60=a;smithi136:172.21.15.136=b;smithi183:172.21.15.183=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ff0104510 con 0x7f5fdc0387a0 2023-07-20T01:22:13.682 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.701+0000 7f5fe7fff700 1 -- 172.21.15.183:0/1174868351 <== mgr.14152 v2:172.21.15.60:6800/1961833180 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f5ff0104510 con 0x7f5fdc0387a0 2023-07-20T01:22:13.682 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled mon update... 2023-07-20T01:22:13.684 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.703+0000 7f5ff5675700 1 -- 172.21.15.183:0/1174868351 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f5fdc0387a0 msgr2=0x7f5fdc03ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:13.684 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.703+0000 7f5ff5675700 1 --2- 172.21.15.183:0/1174868351 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f5fdc0387a0 0x7f5fdc03ac60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5fd8006fd0 tx=0x7f5fd8006e40 comp rx=0 tx=0).stop 2023-07-20T01:22:13.685 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.703+0000 7f5ff5675700 1 -- 172.21.15.183:0/1174868351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ff0105180 msgr2=0x7f5ff0195640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:13.685 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.703+0000 7f5ff5675700 1 --2- 172.21.15.183:0/1174868351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ff0105180 0x7f5ff0195640 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5fe0004f40 tx=0x7f5fe0005020 comp rx=0 tx=0).stop 2023-07-20T01:22:13.685 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.703+0000 7f5ff5675700 1 -- 172.21.15.183:0/1174868351 shutdown_connections 2023-07-20T01:22:13.685 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.703+0000 7f5ff5675700 1 --2- 172.21.15.183:0/1174868351 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f5fdc0387a0 0x7f5fdc03ac60 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:13.685 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.703+0000 7f5ff5675700 1 --2- 172.21.15.183:0/1174868351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ff0105180 0x7f5ff0195640 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:13.685 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.703+0000 7f5ff5675700 1 -- 172.21.15.183:0/1174868351 >> 172.21.15.183:0/1174868351 conn(0x7f5ff0074c70 msgr2=0x7f5ff0102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:22:13.685 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.703+0000 7f5ff5675700 1 -- 172.21.15.183:0/1174868351 shutdown_connections 2023-07-20T01:22:13.686 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:13.703+0000 7f5ff5675700 1 -- 172.21.15.183:0/1174868351 wait complete. 2023-07-20T01:22:14.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:13 smithi060 ceph-mon[96524]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:14.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:13 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:14.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:13 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:22:14.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:13 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:14.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:13 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:22:14.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:13 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:14.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:13 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T01:22:14.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:13 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:14.194 DEBUG:teuthology.orchestra.run.smithi136:mon.b> sudo journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mon.b.service 2023-07-20T01:22:14.197 DEBUG:teuthology.orchestra.run.smithi183:mon.c> sudo journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mon.c.service 2023-07-20T01:22:14.202 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-20T01:22:14.202 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph mon dump -f json 2023-07-20T01:22:14.232 INFO:journalctl@ceph.mon.c.smithi183.stdout:-- Logs begin at Thu 2023-07-20 01:00:08 UTC. -- 2023-07-20T01:22:14.233 INFO:journalctl@ceph.mon.b.smithi136.stdout:-- Logs begin at Thu 2023-07-20 01:00:24 UTC. -- 2023-07-20T01:22:15.022 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T01:22:15.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:14 smithi060 ceph-mon[96524]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi060:172.21.15.60=a;smithi136:172.21.15.136=b;smithi183:172.21.15.183=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:22:15.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:14 smithi060 ceph-mon[96524]: Saving service mon spec with placement smithi060:172.21.15.60=a;smithi136:172.21.15.136=b;smithi183:172.21.15.183=c;count:3 2023-07-20T01:22:15.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:14 smithi060 ceph-mon[96524]: Deploying daemon mon.c on smithi183 2023-07-20T01:22:16.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:15 smithi060 ceph-mon[96524]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:16.858 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:16.874+0000 7f6375915700 1 -- 172.21.15.183:0/2469091815 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6370105180 msgr2=0x7f6370105560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:16.858 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:16.874+0000 7f6375915700 1 --2- 172.21.15.183:0/2469091815 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6370105180 0x7f6370105560 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6360009910 tx=0x7f6360009c20 comp rx=0 tx=0).stop 2023-07-20T01:22:16.858 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:16.874+0000 7f6375915700 1 -- 172.21.15.183:0/2469091815 shutdown_connections 2023-07-20T01:22:16.858 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:16.874+0000 7f6375915700 1 --2- 172.21.15.183:0/2469091815 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6370105180 0x7f6370105560 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:16.858 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:16.874+0000 7f6375915700 1 -- 172.21.15.183:0/2469091815 >> 172.21.15.183:0/2469091815 conn(0x7f6370074c70 msgr2=0x7f6370075080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:22:16.859 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:16.874+0000 7f6375915700 1 -- 172.21.15.183:0/2469091815 shutdown_connections 2023-07-20T01:22:16.859 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:16.874+0000 7f6375915700 1 -- 172.21.15.183:0/2469091815 wait complete. 2023-07-20T01:22:16.859 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:16.875+0000 7f6375915700 1 Processor -- start 2023-07-20T01:22:16.859 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:16.876+0000 7f6375915700 1 -- start start 2023-07-20T01:22:16.859 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:16.876+0000 7f6375915700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6370105180 0x7f63701955d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:22:16.859 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:16.876+0000 7f6375915700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6370195b10 con 0x7f6370105180 2023-07-20T01:22:16.859 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:16.876+0000 7f636effd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6370105180 0x7f63701955d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:22:16.860 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:16.876+0000 7f636effd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6370105180 0x7f63701955d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.183:47646/0 (socket says 172.21.15.183:47646) 2023-07-20T01:22:16.860 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:16.876+0000 7f636effd700 1 -- 172.21.15.183:0/2818592093 learned_addr learned my addr 172.21.15.183:0/2818592093 (peer_addr_for_me v2:172.21.15.183:0/0) 2023-07-20T01:22:16.861 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:16.877+0000 7f636effd700 1 -- 172.21.15.183:0/2818592093 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63600095f0 con 0x7f6370105180 2023-07-20T01:22:16.861 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:16.877+0000 7f636effd700 1 --2- 172.21.15.183:0/2818592093 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6370105180 0x7f63701955d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6360004f40 tx=0x7f6360005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:22:16.861 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:16.877+0000 7f6374913700 1 -- 172.21.15.183:0/2818592093 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f636001d070 con 0x7f6370105180 2023-07-20T01:22:16.861 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:16.877+0000 7f6375915700 1 -- 172.21.15.183:0/2818592093 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6370195d10 con 0x7f6370105180 2023-07-20T01:22:16.861 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:16.877+0000 7f6374913700 1 -- 172.21.15.183:0/2818592093 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f63600044b0 con 0x7f6370105180 2023-07-20T01:22:16.862 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:16.877+0000 7f6374913700 1 -- 172.21.15.183:0/2818592093 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6360003810 con 0x7f6370105180 2023-07-20T01:22:16.862 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:16.877+0000 7f6375915700 1 -- 172.21.15.183:0/2818592093 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6370196190 con 0x7f6370105180 2023-07-20T01:22:16.862 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:16.878+0000 7f6375915700 1 -- 172.21.15.183:0/2818592093 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6370061da0 con 0x7f6370105180 2023-07-20T01:22:16.862 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:16.878+0000 7f6374913700 1 -- 172.21.15.183:0/2818592093 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f6360003ab0 con 0x7f6370105180 2023-07-20T01:22:16.862 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:16.878+0000 7f6374913700 1 --2- 172.21.15.183:0/2818592093 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f635803cbb0 0x7f635803f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:22:16.862 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:16.878+0000 7f6374913700 1 -- 172.21.15.183:0/2818592093 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f636004c670 con 0x7f6370105180 2023-07-20T01:22:16.862 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:16.878+0000 7f636e7fc700 1 --2- 172.21.15.183:0/2818592093 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f635803cbb0 0x7f635803f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:22:16.863 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:16.879+0000 7f636e7fc700 1 --2- 172.21.15.183:0/2818592093 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f635803cbb0 0x7f635803f070 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6364006fd0 tx=0x7f6364006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:22:16.865 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:16.884+0000 7f6374913700 1 -- 172.21.15.183:0/2818592093 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f636002a770 con 0x7f6370105180 2023-07-20T01:22:17.153 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:17.171+0000 7f6375915700 1 -- 172.21.15.183:0/2818592093 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6370196930 con 0x7f6370105180 2023-07-20T01:22:17.154 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:17.172+0000 7f6374913700 1 -- 172.21.15.183:0/2818592093 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+736 (secure 0 0 0) 0x7f636002a360 con 0x7f6370105180 2023-07-20T01:22:17.154 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-07-20T01:22:17.154 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":1,"fsid":"7daf031e-269b-11ee-9b34-001a4aab830c","modified":"2023-07-20T01:20:34.685214Z","created":"2023-07-20T01:20:34.685214Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:3300","nonce":0},{"type":"v1","addr":"172.21.15.60:6789","nonce":0}]},"addr":"172.21.15.60:6789/0","public_addr":"172.21.15.60:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-07-20T01:22:17.156 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:17.174+0000 7f6375915700 1 -- 172.21.15.183:0/2818592093 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f635803cbb0 msgr2=0x7f635803f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:17.156 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:17.174+0000 7f6375915700 1 --2- 172.21.15.183:0/2818592093 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f635803cbb0 0x7f635803f070 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6364006fd0 tx=0x7f6364006e40 comp rx=0 tx=0).stop 2023-07-20T01:22:17.156 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:17.174+0000 7f6375915700 1 -- 172.21.15.183:0/2818592093 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6370105180 msgr2=0x7f63701955d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:17.156 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:17.174+0000 7f6375915700 1 --2- 172.21.15.183:0/2818592093 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6370105180 0x7f63701955d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6360004f40 tx=0x7f6360005020 comp rx=0 tx=0).stop 2023-07-20T01:22:17.156 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:17.174+0000 7f6375915700 1 -- 172.21.15.183:0/2818592093 shutdown_connections 2023-07-20T01:22:17.157 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:17.174+0000 7f6375915700 1 --2- 172.21.15.183:0/2818592093 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f635803cbb0 0x7f635803f070 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:17.157 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:17.174+0000 7f6375915700 1 --2- 172.21.15.183:0/2818592093 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6370105180 0x7f63701955d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:17.157 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:17.174+0000 7f6375915700 1 -- 172.21.15.183:0/2818592093 >> 172.21.15.183:0/2818592093 conn(0x7f6370074c70 msgr2=0x7f63700fd060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:22:17.157 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:17.174+0000 7f6375915700 1 -- 172.21.15.183:0/2818592093 shutdown_connections 2023-07-20T01:22:17.157 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:17.174+0000 7f6375915700 1 -- 172.21.15.183:0/2818592093 wait complete. 2023-07-20T01:22:17.157 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 1 2023-07-20T01:22:17.344 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:17 smithi183 ceph-mon[98477]: expand_channel_meta expanded map: {default=debug} 2023-07-20T01:22:17.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:17 smithi183 systemd[1]: Started Ceph mon.c for 7daf031e-269b-11ee-9b34-001a4aab830c. 2023-07-20T01:22:18.859 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-20T01:22:18.859 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph mon dump -f json 2023-07-20T01:22:19.426 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T01:22:22.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:22 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:22:22.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:22 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:22:22.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:22 smithi060 ceph-mon[96524]: mon.a calling monitor election 2023-07-20T01:22:22.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:22 smithi060 ceph-mon[96524]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:22.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:22 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:22:22.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:22 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:22:22.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:22 smithi060 ceph-mon[96524]: mon.c calling monitor election 2023-07-20T01:22:22.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:22 smithi060 ceph-mon[96524]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:22.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:22 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:22:22.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:22 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:22:22.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:22 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:22:22.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:22 smithi060 ceph-mon[96524]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T01:22:22.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:22 smithi060 ceph-mon[96524]: monmap e2: 2 mons at {a=[v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0],c=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0]} removed_ranks: {} 2023-07-20T01:22:22.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:22 smithi060 ceph-mon[96524]: fsmap 2023-07-20T01:22:22.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:22 smithi060 ceph-mon[96524]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T01:22:22.681 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:22 smithi060 ceph-mon[96524]: mgrmap e13: x(active, since 44s) 2023-07-20T01:22:22.681 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:22 smithi060 ceph-mon[96524]: overall HEALTH_OK 2023-07-20T01:22:22.681 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:22 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:22.681 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:22 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:22.681 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:22 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:22.681 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:22 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T01:22:22.681 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:22 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:22.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:22 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:22:22.702 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:22 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:22:22.702 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:22 smithi183 ceph-mon[98477]: mon.a calling monitor election 2023-07-20T01:22:22.702 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:22 smithi183 ceph-mon[98477]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:22.703 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:22 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:22:22.703 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:22 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:22:22.703 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:22 smithi183 ceph-mon[98477]: mon.c calling monitor election 2023-07-20T01:22:22.703 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:22 smithi183 ceph-mon[98477]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:22.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:22 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:22:22.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:22 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:22:22.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:22 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:22:22.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:22 smithi183 ceph-mon[98477]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T01:22:22.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:22 smithi183 ceph-mon[98477]: monmap e2: 2 mons at {a=[v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0],c=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0]} removed_ranks: {} 2023-07-20T01:22:22.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:22 smithi183 ceph-mon[98477]: fsmap 2023-07-20T01:22:22.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:22 smithi183 ceph-mon[98477]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T01:22:22.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:22 smithi183 ceph-mon[98477]: mgrmap e13: x(active, since 44s) 2023-07-20T01:22:22.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:22 smithi183 ceph-mon[98477]: overall HEALTH_OK 2023-07-20T01:22:22.707 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:22 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:22.707 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:22 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:22.707 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:22 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:22.708 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:22 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T01:22:22.708 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:22 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:23.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:23 smithi060 ceph-mon[96524]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:23.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:23 smithi060 ceph-mon[96524]: Deploying daemon mon.b on smithi136 2023-07-20T01:22:23.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:23 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:22:23.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:23 smithi183 ceph-mon[98477]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:23.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:23 smithi183 ceph-mon[98477]: Deploying daemon mon.b on smithi136 2023-07-20T01:22:23.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:23 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:22:23.706 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.724+0000 7f217362d700 1 -- 172.21.15.183:0/1346418411 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21580035c0 msgr2=0x7f2158005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:23.706 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.724+0000 7f217362d700 1 --2- 172.21.15.183:0/1346418411 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21580035c0 0x7f2158005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f216c0687a0 tx=0x7f215c00feb0 comp rx=0 tx=0).stop 2023-07-20T01:22:23.706 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.724+0000 7f217362d700 1 -- 172.21.15.183:0/1346418411 shutdown_connections 2023-07-20T01:22:23.706 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.724+0000 7f217362d700 1 --2- 172.21.15.183:0/1346418411 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21580035c0 0x7f2158005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:23.706 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.724+0000 7f217362d700 1 -- 172.21.15.183:0/1346418411 >> 172.21.15.183:0/1346418411 conn(0x7f216c0fad70 msgr2=0x7f216c0fd190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:22:23.706 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.724+0000 7f217362d700 1 -- 172.21.15.183:0/1346418411 shutdown_connections 2023-07-20T01:22:23.707 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.724+0000 7f217362d700 1 -- 172.21.15.183:0/1346418411 wait complete. 2023-07-20T01:22:23.708 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.724+0000 7f217362d700 1 Processor -- start 2023-07-20T01:22:23.708 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.724+0000 7f217362d700 1 -- start start 2023-07-20T01:22:23.708 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.724+0000 7f217362d700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21580035c0 0x7f216c19b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:22:23.708 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.724+0000 7f217362d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f216c19baa0 0x7f216c1955e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:22:23.708 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.724+0000 7f217362d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216c19bfe0 con 0x7f216c19baa0 2023-07-20T01:22:23.708 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.724+0000 7f217362d700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216c19c150 con 0x7f21580035c0 2023-07-20T01:22:23.708 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.725+0000 7f21713c9700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21580035c0 0x7f216c19b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:22:23.709 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.725+0000 7f21713c9700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21580035c0 0x7f216c19b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:45798/0 (socket says 172.21.15.183:45798) 2023-07-20T01:22:23.709 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.725+0000 7f21713c9700 1 -- 172.21.15.183:0/3644121800 learned_addr learned my addr 172.21.15.183:0/3644121800 (peer_addr_for_me v2:172.21.15.183:0/0) 2023-07-20T01:22:23.709 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.725+0000 7f21713c9700 1 -- 172.21.15.183:0/3644121800 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21580035c0 msgr2=0x7f216c19b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2023-07-20T01:22:23.709 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.725+0000 7f21713c9700 1 -- 172.21.15.183:0/3644121800 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21580035c0 msgr2=0x7f216c19b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-20T01:22:23.709 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.725+0000 7f21713c9700 1 --2- 172.21.15.183:0/3644121800 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21580035c0 0x7f216c19b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-20T01:22:23.709 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.725+0000 7f21713c9700 1 --2- 172.21.15.183:0/3644121800 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21580035c0 0x7f216c19b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T01:22:23.709 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.725+0000 7f2170bc8700 1 --2- 172.21.15.183:0/3644121800 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f216c19baa0 0x7f216c1955e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:22:23.710 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.725+0000 7f2170bc8700 1 -- 172.21.15.183:0/3644121800 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21580035c0 msgr2=0x7f216c19b560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T01:22:23.710 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.725+0000 7f2170bc8700 1 --2- 172.21.15.183:0/3644121800 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21580035c0 0x7f216c19b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:23.710 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.725+0000 7f2170bc8700 1 -- 172.21.15.183:0/3644121800 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f215c00f900 con 0x7f216c19baa0 2023-07-20T01:22:23.710 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.725+0000 7f2170bc8700 1 --2- 172.21.15.183:0/3644121800 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f216c19baa0 0x7f216c1955e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f216800b500 tx=0x7f216800b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:22:23.711 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.726+0000 7f21627fc700 1 -- 172.21.15.183:0/3644121800 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f2168011630 con 0x7f216c19baa0 2023-07-20T01:22:23.711 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.726+0000 7f21627fc700 1 -- 172.21.15.183:0/3644121800 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f2168011c20 con 0x7f216c19baa0 2023-07-20T01:22:23.711 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.726+0000 7f217362d700 1 -- 172.21.15.183:0/3644121800 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f216c195be0 con 0x7f216c19baa0 2023-07-20T01:22:23.712 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.726+0000 7f21627fc700 1 -- 172.21.15.183:0/3644121800 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f2168010c40 con 0x7f216c19baa0 2023-07-20T01:22:23.712 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.726+0000 7f217362d700 1 -- 172.21.15.183:0/3644121800 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f216c196130 con 0x7f216c19baa0 2023-07-20T01:22:23.712 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.728+0000 7f21627fc700 1 -- 172.21.15.183:0/3644121800 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f216801a410 con 0x7f216c19baa0 2023-07-20T01:22:23.712 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.728+0000 7f21627fc700 1 --2- 172.21.15.183:0/3644121800 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f21500385f0 0x7f215003aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:22:23.712 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.728+0000 7f21627fc700 1 -- 172.21.15.183:0/3644121800 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f216804a870 con 0x7f216c19baa0 2023-07-20T01:22:23.712 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.728+0000 7f217362d700 1 -- 172.21.15.183:0/3644121800 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f216c196460 con 0x7f216c19baa0 2023-07-20T01:22:23.712 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.728+0000 7f21713c9700 1 --2- 172.21.15.183:0/3644121800 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f21500385f0 0x7f215003aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:22:23.712 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.729+0000 7f21713c9700 1 --2- 172.21.15.183:0/3644121800 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f21500385f0 0x7f215003aab0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f215c005b40 tx=0x7f215c005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:22:23.715 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.733+0000 7f21627fc700 1 -- 172.21.15.183:0/3644121800 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2168010da0 con 0x7f216c19baa0 2023-07-20T01:22:23.975 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.993+0000 7f217362d700 1 -- 172.21.15.183:0/3644121800 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f216c061da0 con 0x7f216c19baa0 2023-07-20T01:22:23.976 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.994+0000 7f21627fc700 1 -- 172.21.15.183:0/3644121800 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1006 (secure 0 0 0) 0x7f2168018310 con 0x7f216c19baa0 2023-07-20T01:22:23.976 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-07-20T01:22:23.977 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":2,"fsid":"7daf031e-269b-11ee-9b34-001a4aab830c","modified":"2023-07-20T01:22:17.311539Z","created":"2023-07-20T01:20:34.685214Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:3300","nonce":0},{"type":"v1","addr":"172.21.15.60:6789","nonce":0}]},"addr":"172.21.15.60:6789/0","public_addr":"172.21.15.60:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:3300","nonce":0},{"type":"v1","addr":"172.21.15.183:6789","nonce":0}]},"addr":"172.21.15.183:6789/0","public_addr":"172.21.15.183:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-07-20T01:22:23.978 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.996+0000 7f217362d700 1 -- 172.21.15.183:0/3644121800 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f21500385f0 msgr2=0x7f215003aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:23.979 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.996+0000 7f217362d700 1 --2- 172.21.15.183:0/3644121800 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f21500385f0 0x7f215003aab0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f215c005b40 tx=0x7f215c005a90 comp rx=0 tx=0).stop 2023-07-20T01:22:23.979 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.996+0000 7f217362d700 1 -- 172.21.15.183:0/3644121800 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f216c19baa0 msgr2=0x7f216c1955e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:23.979 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.996+0000 7f217362d700 1 --2- 172.21.15.183:0/3644121800 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f216c19baa0 0x7f216c1955e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f216800b500 tx=0x7f216800b8c0 comp rx=0 tx=0).stop 2023-07-20T01:22:23.979 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.996+0000 7f217362d700 1 -- 172.21.15.183:0/3644121800 shutdown_connections 2023-07-20T01:22:23.980 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.996+0000 7f217362d700 1 --2- 172.21.15.183:0/3644121800 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f21500385f0 0x7f215003aab0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:23.980 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.996+0000 7f217362d700 1 --2- 172.21.15.183:0/3644121800 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21580035c0 0x7f216c19b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:23.980 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.996+0000 7f217362d700 1 --2- 172.21.15.183:0/3644121800 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f216c19baa0 0x7f216c1955e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:23.980 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.996+0000 7f217362d700 1 -- 172.21.15.183:0/3644121800 >> 172.21.15.183:0/3644121800 conn(0x7f216c0fad70 msgr2=0x7f216c102350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:22:23.980 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.996+0000 7f217362d700 1 -- 172.21.15.183:0/3644121800 shutdown_connections 2023-07-20T01:22:23.980 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:23.996+0000 7f217362d700 1 -- 172.21.15.183:0/3644121800 wait complete. 2023-07-20T01:22:23.980 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 2 2023-07-20T01:22:24.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:24 smithi060 ceph-mon[96524]: from='client.? 172.21.15.183:0/3644121800' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T01:22:24.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:24 smithi183 ceph-mon[98477]: from='client.? 172.21.15.183:0/3644121800' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T01:22:25.383 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 systemd[1]: Starting Ceph mon.b for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:22:25.489 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-20T01:22:25.490 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph mon dump -f json 2023-07-20T01:22:25.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 podman[98043]: 2023-07-20T01:22:25.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: set uid:gid to 167:167 (ceph:ceph) 2023-07-20T01:22:25.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: ceph version 18.0.0-5040-g1b0b2eb0 (1b0b2eb005a7c51d97a788b9900d511e1394b537) reef (dev), process ceph-mon, pid 8 2023-07-20T01:22:25.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: pidfile_write: ignore empty --pid-file 2023-07-20T01:22:25.649 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: load: jerasure load: lrc 2023-07-20T01:22:25.649 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: RocksDB version: 7.9.2 2023-07-20T01:22:25.649 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Git sha 0 2023-07-20T01:22:25.649 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Compile date 2023-07-19 21:36:35 2023-07-20T01:22:25.649 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: DB SUMMARY 2023-07-20T01:22:25.649 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: DB Session ID: VBAYCDQWFHPU5UYJUY1J 2023-07-20T01:22:25.649 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: CURRENT file: CURRENT 2023-07-20T01:22:25.649 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: IDENTITY file: IDENTITY 2023-07-20T01:22:25.650 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-07-20T01:22:25.650 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2023-07-20T01:22:25.650 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2023-07-20T01:22:25.650 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.error_if_exists: 0 2023-07-20T01:22:25.650 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.create_if_missing: 0 2023-07-20T01:22:25.650 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.paranoid_checks: 1 2023-07-20T01:22:25.651 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-20T01:22:25.651 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-20T01:22:25.651 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-20T01:22:25.651 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.env: 0x556e0d0b9c60 2023-07-20T01:22:25.651 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.fs: PosixFileSystem 2023-07-20T01:22:25.651 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.info_log: 0x556e0e72b2c0 2023-07-20T01:22:25.652 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.max_file_opening_threads: 16 2023-07-20T01:22:25.652 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.statistics: (nil) 2023-07-20T01:22:25.652 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.use_fsync: 0 2023-07-20T01:22:25.652 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.max_log_file_size: 0 2023-07-20T01:22:25.652 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-20T01:22:25.652 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-20T01:22:25.652 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.keep_log_file_num: 1000 2023-07-20T01:22:25.652 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.recycle_log_file_num: 0 2023-07-20T01:22:25.653 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.allow_fallocate: 1 2023-07-20T01:22:25.653 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.allow_mmap_reads: 0 2023-07-20T01:22:25.653 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.allow_mmap_writes: 0 2023-07-20T01:22:25.653 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.use_direct_reads: 0 2023-07-20T01:22:25.653 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-20T01:22:25.653 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.create_missing_column_families: 0 2023-07-20T01:22:25.653 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.db_log_dir: 2023-07-20T01:22:25.653 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.wal_dir: 2023-07-20T01:22:25.654 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-20T01:22:25.654 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-20T01:22:25.654 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-20T01:22:25.654 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-20T01:22:25.654 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-20T01:22:25.654 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-20T01:22:25.654 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.advise_random_on_open: 1 2023-07-20T01:22:25.655 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.db_write_buffer_size: 0 2023-07-20T01:22:25.655 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.write_buffer_manager: 0x556e0e64c3c0 2023-07-20T01:22:25.655 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-20T01:22:25.655 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-20T01:22:25.655 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-20T01:22:25.655 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.rate_limiter: (nil) 2023-07-20T01:22:25.655 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-20T01:22:25.655 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.wal_recovery_mode: 2 2023-07-20T01:22:25.656 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.enable_thread_tracking: 0 2023-07-20T01:22:25.656 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.enable_pipelined_write: 0 2023-07-20T01:22:25.656 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.unordered_write: 0 2023-07-20T01:22:25.656 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-20T01:22:25.656 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-20T01:22:25.656 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-20T01:22:25.656 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-20T01:22:25.656 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.row_cache: None 2023-07-20T01:22:25.657 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.wal_filter: None 2023-07-20T01:22:25.657 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-20T01:22:25.657 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.allow_ingest_behind: 0 2023-07-20T01:22:25.657 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.two_write_queues: 0 2023-07-20T01:22:25.657 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.manual_wal_flush: 0 2023-07-20T01:22:25.657 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.wal_compression: 0 2023-07-20T01:22:25.657 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.atomic_flush: 0 2023-07-20T01:22:25.657 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-20T01:22:25.658 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-20T01:22:25.658 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-20T01:22:25.658 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.log_readahead_size: 0 2023-07-20T01:22:25.658 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-20T01:22:25.658 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.best_efforts_recovery: 0 2023-07-20T01:22:25.658 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-20T01:22:25.658 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-20T01:22:25.659 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.allow_data_in_errors: 0 2023-07-20T01:22:25.659 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.db_host_id: __hostname__ 2023-07-20T01:22:25.659 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-20T01:22:25.659 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.max_background_jobs: 2 2023-07-20T01:22:25.659 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.max_background_compactions: -1 2023-07-20T01:22:25.659 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.max_subcompactions: 1 2023-07-20T01:22:25.659 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-20T01:22:25.659 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-20T01:22:25.660 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-20T01:22:25.660 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.max_total_wal_size: 0 2023-07-20T01:22:25.660 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-20T01:22:25.660 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-20T01:22:25.660 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-20T01:22:25.660 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-20T01:22:25.660 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.max_open_files: -1 2023-07-20T01:22:25.660 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.bytes_per_sync: 0 2023-07-20T01:22:25.661 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-20T01:22:25.661 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-20T01:22:25.661 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.compaction_readahead_size: 0 2023-07-20T01:22:25.661 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.max_background_flushes: -1 2023-07-20T01:22:25.661 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Compression algorithms supported: 2023-07-20T01:22:25.661 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-20T01:22:25.661 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: kZSTD supported: 0 2023-07-20T01:22:25.662 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: kXpressCompression supported: 0 2023-07-20T01:22:25.662 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: kLZ4HCCompression supported: 1 2023-07-20T01:22:25.662 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: kZlibCompression supported: 1 2023-07-20T01:22:25.662 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: kSnappyCompression supported: 1 2023-07-20T01:22:25.662 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: kLZ4Compression supported: 1 2023-07-20T01:22:25.662 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: kBZip2Compression supported: 0 2023-07-20T01:22:25.662 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-20T01:22:25.662 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-20T01:22:25.663 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2023-07-20T01:22:25.663 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-20T01:22:25.663 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-20T01:22:25.663 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.merge_operator: 2023-07-20T01:22:25.663 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.compaction_filter: None 2023-07-20T01:22:25.663 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.compaction_filter_factory: None 2023-07-20T01:22:25.663 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.sst_partitioner_factory: None 2023-07-20T01:22:25.663 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-20T01:22:25.664 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-20T01:22:25.664 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556e0e72b3c0) 2023-07-20T01:22:25.664 INFO:journalctl@ceph.mon.b.smithi136.stdout: cache_index_and_filter_blocks: 1 2023-07-20T01:22:25.664 INFO:journalctl@ceph.mon.b.smithi136.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-20T01:22:25.665 INFO:journalctl@ceph.mon.b.smithi136.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-20T01:22:25.666 INFO:journalctl@ceph.mon.b.smithi136.stdout: pin_top_level_index_and_filter: 1 2023-07-20T01:22:25.666 INFO:journalctl@ceph.mon.b.smithi136.stdout: index_type: 0 2023-07-20T01:22:25.666 INFO:journalctl@ceph.mon.b.smithi136.stdout: data_block_index_type: 0 2023-07-20T01:22:25.666 INFO:journalctl@ceph.mon.b.smithi136.stdout: index_shortening: 1 2023-07-20T01:22:25.666 INFO:journalctl@ceph.mon.b.smithi136.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-20T01:22:25.666 INFO:journalctl@ceph.mon.b.smithi136.stdout: checksum: 4 2023-07-20T01:22:25.667 INFO:journalctl@ceph.mon.b.smithi136.stdout: no_block_cache: 0 2023-07-20T01:22:25.667 INFO:journalctl@ceph.mon.b.smithi136.stdout: block_cache: 0x556e0e6eb350 2023-07-20T01:22:25.667 INFO:journalctl@ceph.mon.b.smithi136.stdout: block_cache_name: BinnedLRUCache 2023-07-20T01:22:25.667 INFO:journalctl@ceph.mon.b.smithi136.stdout: block_cache_options: 2023-07-20T01:22:25.667 INFO:journalctl@ceph.mon.b.smithi136.stdout: capacity : 536870912 2023-07-20T01:22:25.667 INFO:journalctl@ceph.mon.b.smithi136.stdout: num_shard_bits : 4 2023-07-20T01:22:25.667 INFO:journalctl@ceph.mon.b.smithi136.stdout: strict_capacity_limit : 0 2023-07-20T01:22:25.667 INFO:journalctl@ceph.mon.b.smithi136.stdout: high_pri_pool_ratio: 0.000 2023-07-20T01:22:25.668 INFO:journalctl@ceph.mon.b.smithi136.stdout: block_cache_compressed: (nil) 2023-07-20T01:22:25.668 INFO:journalctl@ceph.mon.b.smithi136.stdout: persistent_cache: (nil) 2023-07-20T01:22:25.668 INFO:journalctl@ceph.mon.b.smithi136.stdout: block_size: 4096 2023-07-20T01:22:25.668 INFO:journalctl@ceph.mon.b.smithi136.stdout: block_size_deviation: 10 2023-07-20T01:22:25.668 INFO:journalctl@ceph.mon.b.smithi136.stdout: block_restart_interval: 16 2023-07-20T01:22:25.668 INFO:journalctl@ceph.mon.b.smithi136.stdout: index_block_restart_interval: 1 2023-07-20T01:22:25.668 INFO:journalctl@ceph.mon.b.smithi136.stdout: metadata_block_size: 4096 2023-07-20T01:22:25.669 INFO:journalctl@ceph.mon.b.smithi136.stdout: partition_filters: 0 2023-07-20T01:22:25.669 INFO:journalctl@ceph.mon.b.smithi136.stdout: use_delta_encoding: 1 2023-07-20T01:22:25.669 INFO:journalctl@ceph.mon.b.smithi136.stdout: filter_policy: bloomfilter 2023-07-20T01:22:25.669 INFO:journalctl@ceph.mon.b.smithi136.stdout: whole_key_filtering: 1 2023-07-20T01:22:25.669 INFO:journalctl@ceph.mon.b.smithi136.stdout: verify_compression: 0 2023-07-20T01:22:25.669 INFO:journalctl@ceph.mon.b.smithi136.stdout: read_amp_bytes_per_bit: 0 2023-07-20T01:22:25.669 INFO:journalctl@ceph.mon.b.smithi136.stdout: format_version: 5 2023-07-20T01:22:25.670 INFO:journalctl@ceph.mon.b.smithi136.stdout: enable_index_compression: 1 2023-07-20T01:22:25.670 INFO:journalctl@ceph.mon.b.smithi136.stdout: block_align: 0 2023-07-20T01:22:25.670 INFO:journalctl@ceph.mon.b.smithi136.stdout: max_auto_readahead_size: 262144 2023-07-20T01:22:25.670 INFO:journalctl@ceph.mon.b.smithi136.stdout: prepopulate_block_cache: 0 2023-07-20T01:22:25.670 INFO:journalctl@ceph.mon.b.smithi136.stdout: initial_auto_readahead_size: 8192 2023-07-20T01:22:25.670 INFO:journalctl@ceph.mon.b.smithi136.stdout: num_file_reads_for_auto_readahead: 2 2023-07-20T01:22:25.670 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.write_buffer_size: 33554432 2023-07-20T01:22:25.670 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.max_write_buffer_number: 2 2023-07-20T01:22:25.671 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.compression: NoCompression 2023-07-20T01:22:25.671 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.bottommost_compression: Disabled 2023-07-20T01:22:25.671 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.prefix_extractor: nullptr 2023-07-20T01:22:25.671 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-20T01:22:25.671 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.num_levels: 7 2023-07-20T01:22:25.671 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-20T01:22:25.671 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-20T01:22:25.672 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-20T01:22:25.672 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-20T01:22:25.672 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-20T01:22:25.672 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-20T01:22:25.672 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-20T01:22:25.672 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-20T01:22:25.672 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-20T01:22:25.672 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-20T01:22:25.673 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-20T01:22:25.673 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-20T01:22:25.673 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-20T01:22:25.673 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.compression_opts.level: 32767 2023-07-20T01:22:25.673 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.compression_opts.strategy: 0 2023-07-20T01:22:25.673 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-20T01:22:25.673 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-20T01:22:25.674 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-20T01:22:25.674 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-20T01:22:25.674 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.compression_opts.enabled: false 2023-07-20T01:22:25.674 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-20T01:22:25.674 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-20T01:22:25.674 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-20T01:22:25.674 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-20T01:22:25.674 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.target_file_size_base: 67108864 2023-07-20T01:22:25.675 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-20T01:22:25.675 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-20T01:22:25.675 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-20T01:22:25.675 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-20T01:22:25.675 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-20T01:22:25.675 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-20T01:22:25.675 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-20T01:22:25.675 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-20T01:22:25.676 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-20T01:22:25.676 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-20T01:22:25.676 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-20T01:22:25.676 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-20T01:22:25.676 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-20T01:22:25.676 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-20T01:22:25.676 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.arena_block_size: 1048576 2023-07-20T01:22:25.676 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-20T01:22:25.677 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-20T01:22:25.677 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.disable_auto_compactions: 0 2023-07-20T01:22:25.677 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-20T01:22:25.677 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-20T01:22:25.677 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-20T01:22:25.677 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-20T01:22:25.678 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-20T01:22:25.678 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-20T01:22:25.678 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-20T01:22:25.678 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-20T01:22:25.678 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-20T01:22:25.678 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-20T01:22:25.678 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.table_properties_collectors: 2023-07-20T01:22:25.679 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.inplace_update_support: 0 2023-07-20T01:22:25.679 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-20T01:22:25.679 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-20T01:22:25.679 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-20T01:22:25.679 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-20T01:22:25.679 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.bloom_locality: 0 2023-07-20T01:22:25.679 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.max_successive_merges: 0 2023-07-20T01:22:25.679 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-20T01:22:25.680 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.paranoid_file_checks: 0 2023-07-20T01:22:25.680 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.force_consistency_checks: 1 2023-07-20T01:22:25.680 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.report_bg_io_stats: 0 2023-07-20T01:22:25.680 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.ttl: 2592000 2023-07-20T01:22:25.680 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-20T01:22:25.680 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-20T01:22:25.680 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-20T01:22:25.680 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.enable_blob_files: false 2023-07-20T01:22:25.681 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.min_blob_size: 0 2023-07-20T01:22:25.681 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.blob_file_size: 268435456 2023-07-20T01:22:25.681 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-20T01:22:25.681 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-20T01:22:25.681 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-20T01:22:25.681 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-20T01:22:25.681 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-20T01:22:25.682 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.blob_file_starting_level: 0 2023-07-20T01:22:25.682 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-20T01:22:25.682 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-20T01:22:25.682 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-07-20T01:22:25.682 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ac379d51-b1d5-4615-a33d-45168b82e612 2023-07-20T01:22:25.682 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689816145622557, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-07-20T01:22:25.682 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-07-20T01:22:25.682 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689816145622985, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689816145, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ac379d51-b1d5-4615-a33d-45168b82e612", "db_session_id": "VBAYCDQWFHPU5UYJUY1J", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-07-20T01:22:25.683 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689816145623052, "job": 1, "event": "recovery_finished"} 2023-07-20T01:22:25.683 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-07-20T01:22:25.683 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-20T01:22:25.683 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x556e0e784000 2023-07-20T01:22:25.683 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: DB pointer 0x556e0e770000 2023-07-20T01:22:25.683 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-20T01:22:25.684 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-20T01:22:25.684 INFO:journalctl@ceph.mon.b.smithi136.stdout: ** DB Stats ** 2023-07-20T01:22:25.684 INFO:journalctl@ceph.mon.b.smithi136.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T01:22:25.684 INFO:journalctl@ceph.mon.b.smithi136.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-20T01:22:25.684 INFO:journalctl@ceph.mon.b.smithi136.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T01:22:25.684 INFO:journalctl@ceph.mon.b.smithi136.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T01:22:25.684 INFO:journalctl@ceph.mon.b.smithi136.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-20T01:22:25.684 INFO:journalctl@ceph.mon.b.smithi136.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-20T01:22:25.685 INFO:journalctl@ceph.mon.b.smithi136.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-20T01:22:25.685 INFO:journalctl@ceph.mon.b.smithi136.stdout: 2023-07-20T01:22:25.685 INFO:journalctl@ceph.mon.b.smithi136.stdout: ** Compaction Stats [default] ** 2023-07-20T01:22:25.685 INFO:journalctl@ceph.mon.b.smithi136.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T01:22:25.685 INFO:journalctl@ceph.mon.b.smithi136.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-20T01:22:25.685 INFO:journalctl@ceph.mon.b.smithi136.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-20T01:22:25.685 INFO:journalctl@ceph.mon.b.smithi136.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-20T01:22:25.686 INFO:journalctl@ceph.mon.b.smithi136.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-20T01:22:25.686 INFO:journalctl@ceph.mon.b.smithi136.stdout: 2023-07-20T01:22:25.686 INFO:journalctl@ceph.mon.b.smithi136.stdout: ** Compaction Stats [default] ** 2023-07-20T01:22:25.686 INFO:journalctl@ceph.mon.b.smithi136.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-20T01:22:25.686 INFO:journalctl@ceph.mon.b.smithi136.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-20T01:22:25.686 INFO:journalctl@ceph.mon.b.smithi136.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-20T01:22:25.686 INFO:journalctl@ceph.mon.b.smithi136.stdout: 2023-07-20T01:22:25.686 INFO:journalctl@ceph.mon.b.smithi136.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-20T01:22:25.687 INFO:journalctl@ceph.mon.b.smithi136.stdout: 2023-07-20T01:22:25.687 INFO:journalctl@ceph.mon.b.smithi136.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-20T01:22:25.687 INFO:journalctl@ceph.mon.b.smithi136.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-20T01:22:25.687 INFO:journalctl@ceph.mon.b.smithi136.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-20T01:22:25.687 INFO:journalctl@ceph.mon.b.smithi136.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-20T01:22:25.687 INFO:journalctl@ceph.mon.b.smithi136.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-20T01:22:25.687 INFO:journalctl@ceph.mon.b.smithi136.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-20T01:22:25.687 INFO:journalctl@ceph.mon.b.smithi136.stdout: Cumulative compaction: 0.00 GB write, 0.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T01:22:25.688 INFO:journalctl@ceph.mon.b.smithi136.stdout: Interval compaction: 0.00 GB write, 0.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-20T01:22:25.688 INFO:journalctl@ceph.mon.b.smithi136.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-20T01:22:25.688 INFO:journalctl@ceph.mon.b.smithi136.stdout: Block cache BinnedLRUCache@0x556e0e6eb350#8 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2023-07-20T01:22:25.688 INFO:journalctl@ceph.mon.b.smithi136.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-07-20T01:22:25.688 INFO:journalctl@ceph.mon.b.smithi136.stdout: 2023-07-20T01:22:25.688 INFO:journalctl@ceph.mon.b.smithi136.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-20T01:22:25.688 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: mon.b does not exist in monmap, will attempt to join an existing cluster 2023-07-20T01:22:25.689 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: using public_addr v2:172.21.15.136:0/0 -> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] 2023-07-20T01:22:25.689 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: starting mon.b rank -1 at public addrs [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] at bind addrs [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 7daf031e-269b-11ee-9b34-001a4aab830c 2023-07-20T01:22:25.689 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: mon.b@-1(???) e0 preinit fsid 7daf031e-269b-11ee-9b34-001a4aab830c 2023-07-20T01:22:25.689 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 bash[98043]: 7ca6ad78c419ed5f61949a6a28a71f95b6c93667ed4a7b907bf2fc0a31a198ea 2023-07-20T01:22:25.689 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: mon.b@-1(synchronizing).mds e1 new map 2023-07-20T01:22:25.689 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: mon.b@-1(synchronizing).mds e1 print_map 2023-07-20T01:22:25.689 INFO:journalctl@ceph.mon.b.smithi136.stdout: e1 2023-07-20T01:22:25.689 INFO:journalctl@ceph.mon.b.smithi136.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-20T01:22:25.690 INFO:journalctl@ceph.mon.b.smithi136.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-20T01:22:25.690 INFO:journalctl@ceph.mon.b.smithi136.stdout: legacy client fscid: -1 2023-07-20T01:22:25.690 INFO:journalctl@ceph.mon.b.smithi136.stdout: 2023-07-20T01:22:25.690 INFO:journalctl@ceph.mon.b.smithi136.stdout: No filesystems configured 2023-07-20T01:22:25.690 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-07-20T01:22:25.690 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-07-20T01:22:25.690 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-07-20T01:22:25.691 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-07-20T01:22:25.691 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-07-20T01:22:25.691 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-07-20T01:22:25.691 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-20T01:22:25.691 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T01:22:25.691 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T01:22:25.691 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-20T01:22:25.691 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:25.692 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:25.692 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T01:22:25.692 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:25.692 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:25.692 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:25.692 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:25.692 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:25.693 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi136", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:22:25.693 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:25.693 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:22:25.693 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: Updating smithi136:/etc/ceph/ceph.conf 2023-07-20T01:22:25.693 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi183", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:22:25.693 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: Updating smithi136:/var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T01:22:25.693 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2023-07-20T01:22:25.693 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: Deploying cephadm binary to smithi183 2023-07-20T01:22:25.694 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:25.694 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:25.694 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:25.694 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: Updating smithi136:/var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T01:22:25.694 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:25.694 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:25.694 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:25.694 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: Added host smithi183 2023-07-20T01:22:25.695 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:22:25.695 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:25.695 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:25.695 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:25.695 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:25.695 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-20T01:22:25.695 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:25.695 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/1820302194' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-20T01:22:25.696 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:25.696 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:25.696 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:25.696 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:25.696 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:22:25.696 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:25.696 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:22:25.697 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: Updating smithi183:/etc/ceph/ceph.conf 2023-07-20T01:22:25.697 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: Updating smithi183:/var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T01:22:25.697 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/1820302194' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-20T01:22:25.697 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T01:22:25.697 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2023-07-20T01:22:25.697 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: Updating smithi183:/var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/config/ceph.client.admin.keyring 2023-07-20T01:22:25.697 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:25.697 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:25.698 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:25.698 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:25.698 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:25.698 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:22:25.698 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:25.698 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:22:25.698 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:25.699 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T01:22:25.699 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:25.699 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi060:172.21.15.60=a;smithi136:172.21.15.136=b;smithi183:172.21.15.183=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:22:25.699 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: Saving service mon spec with placement smithi060:172.21.15.60=a;smithi136:172.21.15.136=b;smithi183:172.21.15.183=c;count:3 2023-07-20T01:22:25.699 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: Deploying daemon mon.c on smithi183 2023-07-20T01:22:25.699 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:25.699 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:22:25.700 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:22:25.700 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: mon.a calling monitor election 2023-07-20T01:22:25.700 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:25.700 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:22:25.700 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:22:25.700 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: mon.c calling monitor election 2023-07-20T01:22:25.700 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:25.701 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:22:25.701 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:22:25.701 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:22:25.701 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T01:22:25.701 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: monmap e2: 2 mons at {a=[v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0],c=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0]} removed_ranks: {} 2023-07-20T01:22:25.701 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: fsmap 2023-07-20T01:22:25.701 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T01:22:25.702 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: mgrmap e13: x(active, since 44s) 2023-07-20T01:22:25.702 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: overall HEALTH_OK 2023-07-20T01:22:25.702 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:25.702 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:25.702 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:25.702 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T01:22:25.702 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:25.702 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:25.703 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: Deploying daemon mon.b on smithi136 2023-07-20T01:22:25.703 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:22:25.703 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: from='client.? 172.21.15.183:0/3644121800' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T01:22:25.703 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:25.703 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-07-20T01:22:25.703 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2023-07-20T01:22:25.703 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: expand_channel_meta expand map: {default=false} 2023-07-20T01:22:25.703 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: expand_channel_meta from 'false' to 'false' 2023-07-20T01:22:25.704 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: expand_channel_meta expanded map: {default=false} 2023-07-20T01:22:25.704 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: expand_channel_meta expand map: {default=info} 2023-07-20T01:22:25.704 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: expand_channel_meta from 'info' to 'info' 2023-07-20T01:22:25.704 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: expand_channel_meta expanded map: {default=info} 2023-07-20T01:22:25.704 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: expand_channel_meta expand map: {default=daemon} 2023-07-20T01:22:25.704 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-20T01:22:25.704 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: expand_channel_meta expanded map: {default=daemon} 2023-07-20T01:22:25.705 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: expand_channel_meta expand map: {default=debug} 2023-07-20T01:22:25.705 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: expand_channel_meta from 'debug' to 'debug' 2023-07-20T01:22:25.705 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 ceph-mon[98083]: expand_channel_meta expanded map: {default=debug} 2023-07-20T01:22:25.705 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:25 smithi136 systemd[1]: Started Ceph mon.b for 7daf031e-269b-11ee-9b34-001a4aab830c. 2023-07-20T01:22:25.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:25 smithi183 ceph-mon[98477]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:25.706 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:25 smithi060 ceph-mon[96524]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:26.053 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T01:22:30.938 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.957+0000 7f533e5f2700 1 -- 172.21.15.183:0/2647644021 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53200035c0 msgr2=0x7f5320005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:30.939 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.957+0000 7f533e5f2700 1 --2- 172.21.15.183:0/2647644021 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53200035c0 0x7f5320005a50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5338076160 tx=0x7f5328014ec0 comp rx=0 tx=0).stop 2023-07-20T01:22:30.939 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.957+0000 7f533e5f2700 1 -- 172.21.15.183:0/2647644021 shutdown_connections 2023-07-20T01:22:30.939 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.957+0000 7f533e5f2700 1 --2- 172.21.15.183:0/2647644021 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53200035c0 0x7f5320005a50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:30.940 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.957+0000 7f533e5f2700 1 -- 172.21.15.183:0/2647644021 >> 172.21.15.183:0/2647644021 conn(0x7f5338074d20 msgr2=0x7f5338075130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:22:30.940 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.957+0000 7f533e5f2700 1 -- 172.21.15.183:0/2647644021 shutdown_connections 2023-07-20T01:22:30.940 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.957+0000 7f533e5f2700 1 -- 172.21.15.183:0/2647644021 wait complete. 2023-07-20T01:22:30.941 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.957+0000 7f533e5f2700 1 Processor -- start 2023-07-20T01:22:30.942 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.958+0000 7f533e5f2700 1 -- start start 2023-07-20T01:22:30.942 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.958+0000 7f533e5f2700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53200035c0 0x7f5338196770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:22:30.942 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.958+0000 7f533e5f2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5338196cb0 0x7f5338197130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:22:30.942 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.958+0000 7f533e5f2700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f533819f7e0 0x7f53381a1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:22:30.942 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.958+0000 7f533e5f2700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53381a2320 con 0x7f5338196cb0 2023-07-20T01:22:30.942 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.958+0000 7f533e5f2700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53381a2490 con 0x7f53200035c0 2023-07-20T01:22:30.942 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.958+0000 7f533e5f2700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53381a2650 con 0x7f533819f7e0 2023-07-20T01:22:30.943 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.958+0000 7f533cb8f700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f533819f7e0 0x7f53381a1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:22:30.943 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.958+0000 7f533cb8f700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f533819f7e0 0x7f53381a1bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:45826/0 (socket says 172.21.15.183:45826) 2023-07-20T01:22:30.943 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.958+0000 7f533cb8f700 1 -- 172.21.15.183:0/1501928582 learned_addr learned my addr 172.21.15.183:0/1501928582 (peer_addr_for_me v2:172.21.15.183:0/0) 2023-07-20T01:22:30.943 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.958+0000 7f53377fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5338196cb0 0x7f5338197130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:22:30.943 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.958+0000 7f5337fff700 1 --2- 172.21.15.183:0/1501928582 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53200035c0 0x7f5338196770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:22:30.943 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.958+0000 7f533cb8f700 1 -- 172.21.15.183:0/1501928582 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f533819f7e0 msgr2=0x7f53381a1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2023-07-20T01:22:30.943 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.958+0000 7f533cb8f700 1 -- 172.21.15.183:0/1501928582 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f533819f7e0 msgr2=0x7f53381a1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-20T01:22:30.943 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.958+0000 7f533cb8f700 1 --2- 172.21.15.183:0/1501928582 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f533819f7e0 0x7f53381a1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-20T01:22:30.944 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.959+0000 7f533cb8f700 1 --2- 172.21.15.183:0/1501928582 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f533819f7e0 0x7f53381a1bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T01:22:30.944 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.959+0000 7f5337fff700 1 -- 172.21.15.183:0/1501928582 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53200035c0 msgr2=0x7f5338196770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2023-07-20T01:22:30.944 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.959+0000 7f5337fff700 1 -- 172.21.15.183:0/1501928582 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53200035c0 msgr2=0x7f5338196770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-20T01:22:30.944 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.959+0000 7f5337fff700 1 --2- 172.21.15.183:0/1501928582 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53200035c0 0x7f5338196770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-20T01:22:30.944 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.959+0000 7f5337fff700 1 --2- 172.21.15.183:0/1501928582 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53200035c0 0x7f5338196770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-20T01:22:30.944 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.959+0000 7f53377fe700 1 -- 172.21.15.183:0/1501928582 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f533819f7e0 msgr2=0x7f53381a1bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T01:22:30.944 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.959+0000 7f53377fe700 1 --2- 172.21.15.183:0/1501928582 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f533819f7e0 0x7f53381a1bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:30.945 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.959+0000 7f53377fe700 1 -- 172.21.15.183:0/1501928582 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53200035c0 msgr2=0x7f5338196770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-20T01:22:30.945 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.959+0000 7f53377fe700 1 --2- 172.21.15.183:0/1501928582 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53200035c0 0x7f5338196770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:30.945 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.959+0000 7f53377fe700 1 -- 172.21.15.183:0/1501928582 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5328014910 con 0x7f5338196cb0 2023-07-20T01:22:30.945 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.959+0000 7f53377fe700 1 --2- 172.21.15.183:0/1501928582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5338196cb0 0x7f5338197130 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f532c00b550 tx=0x7f532c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:22:30.945 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.959+0000 7f53357fa700 1 -- 172.21.15.183:0/1501928582 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532c004230 con 0x7f5338196cb0 2023-07-20T01:22:30.945 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.959+0000 7f533e5f2700 1 -- 172.21.15.183:0/1501928582 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53381a2930 con 0x7f5338196cb0 2023-07-20T01:22:30.945 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.959+0000 7f53357fa700 1 -- 172.21.15.183:0/1501928582 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f532c004390 con 0x7f5338196cb0 2023-07-20T01:22:30.945 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.959+0000 7f53357fa700 1 -- 172.21.15.183:0/1501928582 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532c011a20 con 0x7f5338196cb0 2023-07-20T01:22:30.946 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.959+0000 7f533e5f2700 1 -- 172.21.15.183:0/1501928582 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53381a2e80 con 0x7f5338196cb0 2023-07-20T01:22:30.946 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.960+0000 7f53357fa700 1 -- 172.21.15.183:0/1501928582 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f532c011b80 con 0x7f5338196cb0 2023-07-20T01:22:30.947 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.961+0000 7f533e5f2700 1 -- 172.21.15.183:0/1501928582 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5338061da0 con 0x7f5338196cb0 2023-07-20T01:22:30.947 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.961+0000 7f53357fa700 1 --2- 172.21.15.183:0/1501928582 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f5324038c30 0x7f532403b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:22:30.947 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.961+0000 7f53357fa700 1 -- 172.21.15.183:0/1501928582 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f532c04afa0 con 0x7f5338196cb0 2023-07-20T01:22:30.947 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.961+0000 7f5337fff700 1 --2- 172.21.15.183:0/1501928582 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f5324038c30 0x7f532403b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:22:30.947 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.962+0000 7f5337fff700 1 --2- 172.21.15.183:0/1501928582 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f5324038c30 0x7f532403b0f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5328010010 tx=0x7f532800ff80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:22:30.948 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:30.966+0000 7f53357fa700 1 -- 172.21.15.183:0/1501928582 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f532c01a400 con 0x7f5338196cb0 2023-07-20T01:22:31.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:30 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:22:31.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:30 smithi060 ceph-mon[96524]: mon.a calling monitor election 2023-07-20T01:22:31.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:30 smithi060 ceph-mon[96524]: mon.c calling monitor election 2023-07-20T01:22:31.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:30 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:22:31.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:30 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:22:31.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:30 smithi060 ceph-mon[96524]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:31.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:30 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:22:31.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:30 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:22:31.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:30 smithi060 ceph-mon[96524]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:31.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:30 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:22:31.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:30 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:22:31.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:30 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:22:31.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:30 smithi060 ceph-mon[96524]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T01:22:31.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:30 smithi060 ceph-mon[96524]: monmap e3: 3 mons at {a=[v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0],b=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],c=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0]} removed_ranks: {} 2023-07-20T01:22:31.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:30 smithi060 ceph-mon[96524]: fsmap 2023-07-20T01:22:31.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:30 smithi060 ceph-mon[96524]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T01:22:31.181 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:30 smithi060 ceph-mon[96524]: mgrmap e13: x(active, since 52s) 2023-07-20T01:22:31.181 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:30 smithi060 ceph-mon[96524]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-20T01:22:31.181 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:30 smithi060 ceph-mon[96524]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-20T01:22:31.181 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:30 smithi060 ceph-mon[96524]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-20T01:22:31.181 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:30 smithi060 ceph-mon[96524]: mon.b (rank 2) addr [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] is down (out of quorum) 2023-07-20T01:22:31.181 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:30 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:31.181 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:30 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:31.181 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:30 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:31.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:30 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:22:31.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:30 smithi183 ceph-mon[98477]: mon.a calling monitor election 2023-07-20T01:22:31.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:30 smithi183 ceph-mon[98477]: mon.c calling monitor election 2023-07-20T01:22:31.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:30 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:22:31.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:30 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:22:31.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:30 smithi183 ceph-mon[98477]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:31.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:30 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:22:31.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:30 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:22:31.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:30 smithi183 ceph-mon[98477]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:31.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:30 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:22:31.203 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:30 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:22:31.203 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:30 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:22:31.203 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:30 smithi183 ceph-mon[98477]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T01:22:31.203 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:30 smithi183 ceph-mon[98477]: monmap e3: 3 mons at {a=[v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0],b=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],c=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0]} removed_ranks: {} 2023-07-20T01:22:31.203 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:30 smithi183 ceph-mon[98477]: fsmap 2023-07-20T01:22:31.203 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:30 smithi183 ceph-mon[98477]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T01:22:31.203 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:30 smithi183 ceph-mon[98477]: mgrmap e13: x(active, since 52s) 2023-07-20T01:22:31.203 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:30 smithi183 ceph-mon[98477]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-20T01:22:31.204 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:30 smithi183 ceph-mon[98477]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-20T01:22:31.204 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:30 smithi183 ceph-mon[98477]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-20T01:22:31.204 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:30 smithi183 ceph-mon[98477]: mon.b (rank 2) addr [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] is down (out of quorum) 2023-07-20T01:22:31.204 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:30 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:31.204 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:30 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:31.204 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:30 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:31.225 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:31.244+0000 7f533e5f2700 1 -- 172.21.15.183:0/1501928582 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f53381a3170 con 0x7f5338196cb0 2023-07-20T01:22:31.226 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:31.245+0000 7f53357fa700 1 -- 172.21.15.183:0/1501928582 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1274 (secure 0 0 0) 0x7f532c00f2e0 con 0x7f5338196cb0 2023-07-20T01:22:31.226 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-07-20T01:22:31.226 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":3,"fsid":"7daf031e-269b-11ee-9b34-001a4aab830c","modified":"2023-07-20T01:22:25.652414Z","created":"2023-07-20T01:20:34.685214Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:3300","nonce":0},{"type":"v1","addr":"172.21.15.60:6789","nonce":0}]},"addr":"172.21.15.60:6789/0","public_addr":"172.21.15.60:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:3300","nonce":0},{"type":"v1","addr":"172.21.15.183:6789","nonce":0}]},"addr":"172.21.15.183:6789/0","public_addr":"172.21.15.183:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-07-20T01:22:31.228 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:31.247+0000 7f533e5f2700 1 -- 172.21.15.183:0/1501928582 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f5324038c30 msgr2=0x7f532403b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:31.228 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:31.247+0000 7f533e5f2700 1 --2- 172.21.15.183:0/1501928582 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f5324038c30 0x7f532403b0f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5328010010 tx=0x7f532800ff80 comp rx=0 tx=0).stop 2023-07-20T01:22:31.228 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:31.247+0000 7f533e5f2700 1 -- 172.21.15.183:0/1501928582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5338196cb0 msgr2=0x7f5338197130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:31.228 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:31.247+0000 7f533e5f2700 1 --2- 172.21.15.183:0/1501928582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5338196cb0 0x7f5338197130 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f532c00b550 tx=0x7f532c00b910 comp rx=0 tx=0).stop 2023-07-20T01:22:31.229 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:31.247+0000 7f533e5f2700 1 -- 172.21.15.183:0/1501928582 shutdown_connections 2023-07-20T01:22:31.229 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:31.247+0000 7f533e5f2700 1 --2- 172.21.15.183:0/1501928582 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f5324038c30 0x7f532403b0f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:31.229 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:31.247+0000 7f533e5f2700 1 --2- 172.21.15.183:0/1501928582 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f533819f7e0 0x7f53381a1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:31.229 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:31.247+0000 7f533e5f2700 1 --2- 172.21.15.183:0/1501928582 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53200035c0 0x7f5338196770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:31.229 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:31.247+0000 7f533e5f2700 1 --2- 172.21.15.183:0/1501928582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5338196cb0 0x7f5338197130 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:31.229 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:31.247+0000 7f533e5f2700 1 -- 172.21.15.183:0/1501928582 >> 172.21.15.183:0/1501928582 conn(0x7f5338074d20 msgr2=0x7f53380fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:22:31.229 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:31.247+0000 7f533e5f2700 1 -- 172.21.15.183:0/1501928582 shutdown_connections 2023-07-20T01:22:31.229 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:31.247+0000 7f533e5f2700 1 -- 172.21.15.183:0/1501928582 wait complete. 2023-07-20T01:22:31.230 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 3 2023-07-20T01:22:31.613 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-07-20T01:22:31.614 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph config generate-minimal-conf 2023-07-20T01:22:31.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:31 smithi060 ceph-mon[96524]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:31.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:31 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:31.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:31 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:22:31.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:31 smithi060 ceph-mon[96524]: from='client.? 172.21.15.183:0/1501928582' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T01:22:31.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:31 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:22:31.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:31 smithi183 ceph-mon[98477]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:31.952 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:31 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:31.952 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:31 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:22:31.952 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:31 smithi183 ceph-mon[98477]: from='client.? 172.21.15.183:0/1501928582' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T01:22:31.953 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:31 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:22:32.191 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:22:32.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:32 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:22:32.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:32 smithi136 ceph-mon[98083]: mon.a calling monitor election 2023-07-20T01:22:32.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:32 smithi136 ceph-mon[98083]: mon.c calling monitor election 2023-07-20T01:22:32.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:32 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:22:32.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:32 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:22:32.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:32 smithi136 ceph-mon[98083]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:32.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:32 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:22:32.900 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:32 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:22:32.900 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:32 smithi136 ceph-mon[98083]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:32.900 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:32 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:22:32.900 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:32 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:22:32.900 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:32 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:22:32.900 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:32 smithi136 ceph-mon[98083]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-20T01:22:32.900 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:32 smithi136 ceph-mon[98083]: monmap e3: 3 mons at {a=[v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0],b=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],c=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0]} removed_ranks: {} 2023-07-20T01:22:32.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:32 smithi136 ceph-mon[98083]: fsmap 2023-07-20T01:22:32.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:32 smithi136 ceph-mon[98083]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T01:22:32.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:32 smithi136 ceph-mon[98083]: mgrmap e13: x(active, since 52s) 2023-07-20T01:22:32.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:32 smithi136 ceph-mon[98083]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-20T01:22:32.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:32 smithi136 ceph-mon[98083]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-20T01:22:32.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:32 smithi136 ceph-mon[98083]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-20T01:22:32.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:32 smithi136 ceph-mon[98083]: mon.b (rank 2) addr [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] is down (out of quorum) 2023-07-20T01:22:32.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:32 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:32.902 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:32 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:32.902 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:32 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:32.902 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:32 smithi136 ceph-mon[98083]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:32.902 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:32 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:32.902 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:32 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:22:32.902 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:32 smithi136 ceph-mon[98083]: from='client.? 172.21.15.183:0/1501928582' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-20T01:22:32.902 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:32 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:22:33.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.612+0000 7f9bd62d9700 1 -- 172.21.15.60:0/2875714426 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bd01040e0 msgr2=0x7f9bd01044c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:33.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.612+0000 7f9bd62d9700 1 --2- 172.21.15.60:0/2875714426 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bd01040e0 0x7f9bd01044c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0009910 tx=0x7f9bc0009c20 comp rx=0 tx=0).stop 2023-07-20T01:22:33.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.613+0000 7f9bd62d9700 1 -- 172.21.15.60:0/2875714426 shutdown_connections 2023-07-20T01:22:33.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.613+0000 7f9bd62d9700 1 --2- 172.21.15.60:0/2875714426 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bd01040e0 0x7f9bd01044c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:33.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.613+0000 7f9bd62d9700 1 -- 172.21.15.60:0/2875714426 >> 172.21.15.60:0/2875714426 conn(0x7f9bd00757d0 msgr2=0x7f9bd0075be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:22:33.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.613+0000 7f9bd62d9700 1 -- 172.21.15.60:0/2875714426 shutdown_connections 2023-07-20T01:22:33.595 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.613+0000 7f9bd62d9700 1 -- 172.21.15.60:0/2875714426 wait complete. 2023-07-20T01:22:33.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.613+0000 7f9bd62d9700 1 Processor -- start 2023-07-20T01:22:33.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.614+0000 7f9bd62d9700 1 -- start start 2023-07-20T01:22:33.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.614+0000 7f9bd62d9700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9bd01040e0 0x7f9bd0196660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:22:33.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.614+0000 7f9bd62d9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bd0196ba0 0x7f9bd0197020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:22:33.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.614+0000 7f9bd62d9700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bd019f6d0 0x7f9bd01a1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:22:33.597 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.614+0000 7f9bd62d9700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd01a2210 con 0x7f9bd0196ba0 2023-07-20T01:22:33.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.614+0000 7f9bd62d9700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd01a2380 con 0x7f9bd01040e0 2023-07-20T01:22:33.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.614+0000 7f9bd62d9700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd01a2540 con 0x7f9bd019f6d0 2023-07-20T01:22:33.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.614+0000 7f9bcf7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bd0196ba0 0x7f9bd0197020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:22:33.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.614+0000 7f9bcf7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bd0196ba0 0x7f9bd0197020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42210/0 (socket says 172.21.15.60:42210) 2023-07-20T01:22:33.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.614+0000 7f9bcf7fe700 1 -- 172.21.15.60:0/2334264257 learned_addr learned my addr 172.21.15.60:0/2334264257 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:22:33.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.614+0000 7f9bd4876700 1 --2- 172.21.15.60:0/2334264257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bd019f6d0 0x7f9bd01a1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:22:33.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.614+0000 7f9bcf7fe700 1 -- 172.21.15.60:0/2334264257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bd019f6d0 msgr2=0x7f9bd01a1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:33.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.614+0000 7f9bcf7fe700 1 --2- 172.21.15.60:0/2334264257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bd019f6d0 0x7f9bd01a1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:33.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.614+0000 7f9bcf7fe700 1 -- 172.21.15.60:0/2334264257 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9bd01040e0 msgr2=0x7f9bd0196660 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-07-20T01:22:33.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.614+0000 7f9bcf7fe700 1 --2- 172.21.15.60:0/2334264257 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9bd01040e0 0x7f9bd0196660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:33.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.615+0000 7f9bcf7fe700 1 -- 172.21.15.60:0/2334264257 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc00095f0 con 0x7f9bd0196ba0 2023-07-20T01:22:33.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.615+0000 7f9bcf7fe700 1 --2- 172.21.15.60:0/2334264257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bd0196ba0 0x7f9bd0197020 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9bc400b780 tx=0x7f9bc400bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:22:33.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.615+0000 7f9bcd7fa700 1 -- 172.21.15.60:0/2334264257 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc4004230 con 0x7f9bd0196ba0 2023-07-20T01:22:33.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.615+0000 7f9bcd7fa700 1 -- 172.21.15.60:0/2334264257 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9bc4004390 con 0x7f9bd0196ba0 2023-07-20T01:22:33.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.615+0000 7f9bd62d9700 1 -- 172.21.15.60:0/2334264257 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd01a2820 con 0x7f9bd0196ba0 2023-07-20T01:22:33.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.615+0000 7f9bcd7fa700 1 -- 172.21.15.60:0/2334264257 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc4011a60 con 0x7f9bd0196ba0 2023-07-20T01:22:33.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.615+0000 7f9bd62d9700 1 -- 172.21.15.60:0/2334264257 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd01a2d20 con 0x7f9bd0196ba0 2023-07-20T01:22:33.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.616+0000 7f9bcd7fa700 1 -- 172.21.15.60:0/2334264257 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f9bc4011c00 con 0x7f9bd0196ba0 2023-07-20T01:22:33.601 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.616+0000 7f9bcd7fa700 1 --2- 172.21.15.60:0/2334264257 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f9bb8038c80 0x7f9bb803b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:22:33.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.616+0000 7f9bcd7fa700 1 -- 172.21.15.60:0/2334264257 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9bc404af70 con 0x7f9bd0196ba0 2023-07-20T01:22:33.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.616+0000 7f9bd62d9700 1 -- 172.21.15.60:0/2334264257 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bd0061da0 con 0x7f9bd0196ba0 2023-07-20T01:22:33.602 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.616+0000 7f9bcffff700 1 --2- 172.21.15.60:0/2334264257 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f9bb8038c80 0x7f9bb803b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:22:33.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.617+0000 7f9bcffff700 1 --2- 172.21.15.60:0/2334264257 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f9bb8038c80 0x7f9bb803b140 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0005cb0 tx=0x7f9bc0005c20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:22:33.603 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.621+0000 7f9bcd7fa700 1 -- 172.21.15.60:0/2334264257 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9bc401a400 con 0x7f9bd0196ba0 2023-07-20T01:22:33.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:33 smithi060 ceph-mon[96524]: mon.b calling monitor election 2023-07-20T01:22:33.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:33 smithi060 ceph-mon[96524]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:33.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:33 smithi060 ceph-mon[96524]: Updating smithi060:/etc/ceph/ceph.conf 2023-07-20T01:22:33.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:33 smithi060 ceph-mon[96524]: Updating smithi136:/etc/ceph/ceph.conf 2023-07-20T01:22:33.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:33 smithi060 ceph-mon[96524]: Updating smithi183:/etc/ceph/ceph.conf 2023-07-20T01:22:33.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:33 smithi060 ceph-mon[96524]: mon.b calling monitor election 2023-07-20T01:22:33.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:33 smithi060 ceph-mon[96524]: mon.c calling monitor election 2023-07-20T01:22:33.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:33 smithi060 ceph-mon[96524]: mon.a calling monitor election 2023-07-20T01:22:33.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:33 smithi060 ceph-mon[96524]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-20T01:22:33.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:33 smithi060 ceph-mon[96524]: monmap e3: 3 mons at {a=[v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0],b=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],c=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0]} removed_ranks: {} 2023-07-20T01:22:33.681 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:33 smithi060 ceph-mon[96524]: fsmap 2023-07-20T01:22:33.681 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:33 smithi060 ceph-mon[96524]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T01:22:33.681 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:33 smithi060 ceph-mon[96524]: mgrmap e13: x(active, since 54s) 2023-07-20T01:22:33.681 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:33 smithi060 ceph-mon[96524]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-20T01:22:33.681 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:33 smithi060 ceph-mon[96524]: Cluster is now healthy 2023-07-20T01:22:33.681 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:33 smithi060 ceph-mon[96524]: overall HEALTH_OK 2023-07-20T01:22:33.681 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:33 smithi060 ceph-mon[96524]: Updating smithi183:/var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T01:22:33.682 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:33 smithi060 ceph-mon[96524]: Updating smithi136:/var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T01:22:33.682 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:33 smithi060 ceph-mon[96524]: Updating smithi060:/var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T01:22:33.682 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:33 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:33.682 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:33 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:33.682 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:33 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:33.683 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:33 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:33.683 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:33 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:33.683 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:33 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:33.683 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:33 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:33.683 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:33 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:33.683 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:33 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T01:22:33.684 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:33 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T01:22:33.684 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:33 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:33.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:33 smithi183 ceph-mon[98477]: mon.b calling monitor election 2023-07-20T01:22:33.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:33 smithi183 ceph-mon[98477]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:33.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:33 smithi183 ceph-mon[98477]: Updating smithi060:/etc/ceph/ceph.conf 2023-07-20T01:22:33.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:33 smithi183 ceph-mon[98477]: Updating smithi136:/etc/ceph/ceph.conf 2023-07-20T01:22:33.702 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:33 smithi183 ceph-mon[98477]: Updating smithi183:/etc/ceph/ceph.conf 2023-07-20T01:22:33.702 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:33 smithi183 ceph-mon[98477]: mon.b calling monitor election 2023-07-20T01:22:33.702 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:33 smithi183 ceph-mon[98477]: mon.c calling monitor election 2023-07-20T01:22:33.702 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:33 smithi183 ceph-mon[98477]: mon.a calling monitor election 2023-07-20T01:22:33.702 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:33 smithi183 ceph-mon[98477]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-20T01:22:33.702 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:33 smithi183 ceph-mon[98477]: monmap e3: 3 mons at {a=[v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0],b=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],c=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0]} removed_ranks: {} 2023-07-20T01:22:33.702 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:33 smithi183 ceph-mon[98477]: fsmap 2023-07-20T01:22:33.703 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:33 smithi183 ceph-mon[98477]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T01:22:33.703 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:33 smithi183 ceph-mon[98477]: mgrmap e13: x(active, since 54s) 2023-07-20T01:22:33.703 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:33 smithi183 ceph-mon[98477]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-20T01:22:33.703 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:33 smithi183 ceph-mon[98477]: Cluster is now healthy 2023-07-20T01:22:33.703 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:33 smithi183 ceph-mon[98477]: overall HEALTH_OK 2023-07-20T01:22:33.703 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:33 smithi183 ceph-mon[98477]: Updating smithi183:/var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T01:22:33.703 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:33 smithi183 ceph-mon[98477]: Updating smithi136:/var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T01:22:33.703 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:33 smithi183 ceph-mon[98477]: Updating smithi060:/var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T01:22:33.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:33 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:33.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:33 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:33.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:33 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:33.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:33 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:33.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:33 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:33.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:33 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:33.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:33 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:33.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:33 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:33.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:33 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T01:22:33.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:33 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T01:22:33.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:33 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:33.811 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.829+0000 7f9bd62d9700 1 -- 172.21.15.60:0/2334264257 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f9bd01a30d0 con 0x7f9bd0196ba0 2023-07-20T01:22:33.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.830+0000 7f9bcd7fa700 1 -- 172.21.15.60:0/2334264257 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+275 (secure 0 0 0) 0x7f9bc403d030 con 0x7f9bd0196ba0 2023-07-20T01:22:33.812 INFO:teuthology.orchestra.run.smithi060.stdout:# minimal ceph.conf for 7daf031e-269b-11ee-9b34-001a4aab830c 2023-07-20T01:22:33.813 INFO:teuthology.orchestra.run.smithi060.stdout:[global] 2023-07-20T01:22:33.813 INFO:teuthology.orchestra.run.smithi060.stdout: fsid = 7daf031e-269b-11ee-9b34-001a4aab830c 2023-07-20T01:22:33.813 INFO:teuthology.orchestra.run.smithi060.stdout: mon_host = [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2023-07-20T01:22:33.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.832+0000 7f9bd62d9700 1 -- 172.21.15.60:0/2334264257 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f9bb8038c80 msgr2=0x7f9bb803b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:33.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.832+0000 7f9bd62d9700 1 --2- 172.21.15.60:0/2334264257 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f9bb8038c80 0x7f9bb803b140 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0005cb0 tx=0x7f9bc0005c20 comp rx=0 tx=0).stop 2023-07-20T01:22:33.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.832+0000 7f9bd62d9700 1 -- 172.21.15.60:0/2334264257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bd0196ba0 msgr2=0x7f9bd0197020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:33.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.832+0000 7f9bd62d9700 1 --2- 172.21.15.60:0/2334264257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bd0196ba0 0x7f9bd0197020 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9bc400b780 tx=0x7f9bc400bb40 comp rx=0 tx=0).stop 2023-07-20T01:22:33.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.832+0000 7f9bd62d9700 1 -- 172.21.15.60:0/2334264257 shutdown_connections 2023-07-20T01:22:33.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.832+0000 7f9bd62d9700 1 --2- 172.21.15.60:0/2334264257 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f9bb8038c80 0x7f9bb803b140 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:33.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.832+0000 7f9bd62d9700 1 --2- 172.21.15.60:0/2334264257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bd019f6d0 0x7f9bd01a1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:33.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.832+0000 7f9bd62d9700 1 --2- 172.21.15.60:0/2334264257 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9bd01040e0 0x7f9bd0196660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:33.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.832+0000 7f9bd62d9700 1 --2- 172.21.15.60:0/2334264257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bd0196ba0 0x7f9bd0197020 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:33.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.832+0000 7f9bd62d9700 1 -- 172.21.15.60:0/2334264257 >> 172.21.15.60:0/2334264257 conn(0x7f9bd00757d0 msgr2=0x7f9bd0100d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:22:33.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.832+0000 7f9bd62d9700 1 -- 172.21.15.60:0/2334264257 shutdown_connections 2023-07-20T01:22:33.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:33.832+0000 7f9bd62d9700 1 -- 172.21.15.60:0/2334264257 wait complete. 2023-07-20T01:22:33.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:33 smithi136 ceph-mon[98083]: mon.b calling monitor election 2023-07-20T01:22:33.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:33 smithi136 ceph-mon[98083]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:33.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:33 smithi136 ceph-mon[98083]: Updating smithi060:/etc/ceph/ceph.conf 2023-07-20T01:22:33.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:33 smithi136 ceph-mon[98083]: Updating smithi136:/etc/ceph/ceph.conf 2023-07-20T01:22:33.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:33 smithi136 ceph-mon[98083]: Updating smithi183:/etc/ceph/ceph.conf 2023-07-20T01:22:33.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:33 smithi136 ceph-mon[98083]: mon.b calling monitor election 2023-07-20T01:22:33.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:33 smithi136 ceph-mon[98083]: mon.c calling monitor election 2023-07-20T01:22:33.900 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:33 smithi136 ceph-mon[98083]: mon.a calling monitor election 2023-07-20T01:22:33.900 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:33 smithi136 ceph-mon[98083]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-20T01:22:33.900 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:33 smithi136 ceph-mon[98083]: monmap e3: 3 mons at {a=[v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0],b=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],c=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0]} removed_ranks: {} 2023-07-20T01:22:33.900 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:33 smithi136 ceph-mon[98083]: fsmap 2023-07-20T01:22:33.900 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:33 smithi136 ceph-mon[98083]: osdmap e4: 0 total, 0 up, 0 in 2023-07-20T01:22:33.900 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:33 smithi136 ceph-mon[98083]: mgrmap e13: x(active, since 54s) 2023-07-20T01:22:33.900 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:33 smithi136 ceph-mon[98083]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-20T01:22:33.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:33 smithi136 ceph-mon[98083]: Cluster is now healthy 2023-07-20T01:22:33.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:33 smithi136 ceph-mon[98083]: overall HEALTH_OK 2023-07-20T01:22:33.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:33 smithi136 ceph-mon[98083]: Updating smithi183:/var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T01:22:33.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:33 smithi136 ceph-mon[98083]: Updating smithi136:/var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T01:22:33.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:33 smithi136 ceph-mon[98083]: Updating smithi060:/var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/config/ceph.conf 2023-07-20T01:22:33.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:33 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:33.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:33 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:33.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:33 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:33.902 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:33 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:33.902 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:33 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:33.902 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:33 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:33.902 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:33 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:33.902 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:33 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:33.902 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:33 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T01:22:33.902 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:33 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T01:22:33.902 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:33 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:34.344 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-07-20T01:22:34.345 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-07-20T01:22:34.345 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/ceph/ceph.conf 2023-07-20T01:22:34.376 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-07-20T01:22:34.377 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T01:22:34.445 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-07-20T01:22:34.446 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.conf 2023-07-20T01:22:34.482 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-07-20T01:22:34.482 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T01:22:34.552 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2023-07-20T01:22:34.553 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.conf 2023-07-20T01:22:34.589 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2023-07-20T01:22:34.589 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-20T01:22:34.594 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:34 smithi060 ceph-mon[96524]: Reconfiguring mon.a (unknown last config time)... 2023-07-20T01:22:34.594 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:34 smithi060 ceph-mon[96524]: Reconfiguring daemon mon.a on smithi060 2023-07-20T01:22:34.594 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:34 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:22:34.595 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:34 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/2334264257' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:34.659 INFO:tasks.cephadm:Adding mgr.x on smithi060 2023-07-20T01:22:34.660 INFO:tasks.cephadm:Adding mgr.y on smithi136 2023-07-20T01:22:34.660 INFO:tasks.cephadm:Adding mgr.z on smithi183 2023-07-20T01:22:34.660 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph orch apply mgr '3;smithi060=x;smithi136=y;smithi183=z' 2023-07-20T01:22:34.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:34 smithi183 ceph-mon[98477]: Reconfiguring mon.a (unknown last config time)... 2023-07-20T01:22:34.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:34 smithi183 ceph-mon[98477]: Reconfiguring daemon mon.a on smithi060 2023-07-20T01:22:34.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:34 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:22:34.702 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:34 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/2334264257' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:34.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:34 smithi136 ceph-mon[98083]: Reconfiguring mon.a (unknown last config time)... 2023-07-20T01:22:34.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:34 smithi136 ceph-mon[98083]: Reconfiguring daemon mon.a on smithi060 2023-07-20T01:22:34.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:34 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:22:34.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:34 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/2334264257' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:35.252 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T01:22:35.739 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:35 smithi183 ceph-mon[98477]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:35.739 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:35 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:35.739 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:35 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:35.739 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:35 smithi183 ceph-mon[98477]: Reconfiguring mon.b (monmap changed)... 2023-07-20T01:22:35.739 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:35 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T01:22:35.740 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:35 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T01:22:35.740 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:35 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:35.740 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:35 smithi183 ceph-mon[98477]: Reconfiguring daemon mon.b on smithi136 2023-07-20T01:22:35.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:35 smithi136 ceph-mon[98083]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:35.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:35 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:35.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:35 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:35.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:35 smithi136 ceph-mon[98083]: Reconfiguring mon.b (monmap changed)... 2023-07-20T01:22:35.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:35 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T01:22:35.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:35 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T01:22:35.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:35 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:35.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:35 smithi136 ceph-mon[98083]: Reconfiguring daemon mon.b on smithi136 2023-07-20T01:22:36.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:35 smithi060 ceph-mon[96524]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:36.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:35 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:36.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:35 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:36.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:35 smithi060 ceph-mon[96524]: Reconfiguring mon.b (monmap changed)... 2023-07-20T01:22:36.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:35 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T01:22:36.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:35 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T01:22:36.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:35 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:36.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:35 smithi060 ceph-mon[96524]: Reconfiguring daemon mon.b on smithi136 2023-07-20T01:22:36.887 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.906+0000 7fad39b0b700 1 -- 172.21.15.183:0/230544129 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad340ff280 msgr2=0x7fad340ff6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:36.888 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.906+0000 7fad39b0b700 1 --2- 172.21.15.183:0/230544129 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad340ff280 0x7fad340ff6a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fad24009910 tx=0x7fad24009c20 comp rx=0 tx=0).stop 2023-07-20T01:22:36.888 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.906+0000 7fad39b0b700 1 -- 172.21.15.183:0/230544129 shutdown_connections 2023-07-20T01:22:36.888 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.906+0000 7fad39b0b700 1 --2- 172.21.15.183:0/230544129 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad340ff280 0x7fad340ff6a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:36.888 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.906+0000 7fad39b0b700 1 -- 172.21.15.183:0/230544129 >> 172.21.15.183:0/230544129 conn(0x7fad340fa840 msgr2=0x7fad340fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:22:36.888 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.906+0000 7fad39b0b700 1 -- 172.21.15.183:0/230544129 shutdown_connections 2023-07-20T01:22:36.889 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.906+0000 7fad39b0b700 1 -- 172.21.15.183:0/230544129 wait complete. 2023-07-20T01:22:36.889 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.906+0000 7fad39b0b700 1 Processor -- start 2023-07-20T01:22:36.891 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.907+0000 7fad39b0b700 1 -- start start 2023-07-20T01:22:36.891 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.907+0000 7fad39b0b700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fad340ff280 0x7fad34195570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:22:36.891 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.907+0000 7fad39b0b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad34195ab0 0x7fad3419ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:22:36.891 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.907+0000 7fad39b0b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fad34195f30 0x7fad3419d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:22:36.891 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.907+0000 7fad39b0b700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3419d850 con 0x7fad34195ab0 2023-07-20T01:22:36.891 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.907+0000 7fad39b0b700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3419d9c0 con 0x7fad34195f30 2023-07-20T01:22:36.892 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.907+0000 7fad39b0b700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3419db80 con 0x7fad340ff280 2023-07-20T01:22:36.892 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.907+0000 7fad337fe700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fad340ff280 0x7fad34195570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:22:36.892 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.907+0000 7fad337fe700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fad340ff280 0x7fad34195570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:45832/0 (socket says 172.21.15.183:45832) 2023-07-20T01:22:36.892 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.907+0000 7fad337fe700 1 -- 172.21.15.183:0/2780441124 learned_addr learned my addr 172.21.15.183:0/2780441124 (peer_addr_for_me v2:172.21.15.183:0/0) 2023-07-20T01:22:36.892 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.907+0000 7fad32ffd700 1 --2- 172.21.15.183:0/2780441124 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad34195ab0 0x7fad3419ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:22:36.892 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.907+0000 7fad33fff700 1 --2- 172.21.15.183:0/2780441124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fad34195f30 0x7fad3419d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:22:36.892 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.907+0000 7fad337fe700 1 -- 172.21.15.183:0/2780441124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fad34195f30 msgr2=0x7fad3419d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:36.892 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.907+0000 7fad337fe700 1 --2- 172.21.15.183:0/2780441124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fad34195f30 0x7fad3419d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:36.893 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.907+0000 7fad337fe700 1 -- 172.21.15.183:0/2780441124 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad34195ab0 msgr2=0x7fad3419ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:36.893 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.907+0000 7fad337fe700 1 --2- 172.21.15.183:0/2780441124 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad34195ab0 0x7fad3419ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:36.893 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.907+0000 7fad337fe700 1 -- 172.21.15.183:0/2780441124 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad240095f0 con 0x7fad340ff280 2023-07-20T01:22:36.893 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.907+0000 7fad337fe700 1 --2- 172.21.15.183:0/2780441124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fad340ff280 0x7fad34195570 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fad24000c00 tx=0x7fad2400bcf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:22:36.893 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.907+0000 7fad30ff9700 1 -- 172.21.15.183:0/2780441124 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2401d070 con 0x7fad340ff280 2023-07-20T01:22:36.893 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.908+0000 7fad39b0b700 1 -- 172.21.15.183:0/2780441124 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad3419de00 con 0x7fad340ff280 2023-07-20T01:22:36.893 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.908+0000 7fad30ff9700 1 -- 172.21.15.183:0/2780441124 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fad24004db0 con 0x7fad340ff280 2023-07-20T01:22:36.894 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.908+0000 7fad30ff9700 1 -- 172.21.15.183:0/2780441124 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad24022da0 con 0x7fad340ff280 2023-07-20T01:22:36.894 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.908+0000 7fad39b0b700 1 -- 172.21.15.183:0/2780441124 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad3419e230 con 0x7fad340ff280 2023-07-20T01:22:36.894 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.908+0000 7fad30ff9700 1 -- 172.21.15.183:0/2780441124 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fad2400f460 con 0x7fad340ff280 2023-07-20T01:22:36.894 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.908+0000 7fad30ff9700 1 --2- 172.21.15.183:0/2780441124 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fad20038c30 0x7fad2003b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:22:36.894 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.908+0000 7fad30ff9700 1 -- 172.21.15.183:0/2780441124 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fad2404d630 con 0x7fad340ff280 2023-07-20T01:22:36.894 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.909+0000 7fad32ffd700 1 --2- 172.21.15.183:0/2780441124 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fad20038c30 0x7fad2003b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:22:36.894 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.909+0000 7fad39b0b700 1 -- 172.21.15.183:0/2780441124 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad3418ec60 con 0x7fad340ff280 2023-07-20T01:22:36.894 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.909+0000 7fad32ffd700 1 --2- 172.21.15.183:0/2780441124 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fad20038c30 0x7fad2003b0f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fad1c009ad0 tx=0x7fad1c009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:22:36.895 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:36.914+0000 7fad30ff9700 1 -- 172.21.15.183:0/2780441124 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fad24027030 con 0x7fad340ff280 2023-07-20T01:22:37.094 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:37.112+0000 7fad39b0b700 1 -- 172.21.15.183:0/2780441124 --> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi060=x;smithi136=y;smithi183=z", "target": ["mon-mgr", ""]}) v1 -- 0x7fad34060bb0 con 0x7fad20038c30 2023-07-20T01:22:37.100 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:37.119+0000 7fad30ff9700 1 -- 172.21.15.183:0/2780441124 <== mgr.14152 v2:172.21.15.60:6800/1961833180 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fad34060bb0 con 0x7fad20038c30 2023-07-20T01:22:37.101 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled mgr update... 2023-07-20T01:22:37.103 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:37.121+0000 7fad39b0b700 1 -- 172.21.15.183:0/2780441124 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fad20038c30 msgr2=0x7fad2003b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:37.103 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:37.121+0000 7fad39b0b700 1 --2- 172.21.15.183:0/2780441124 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fad20038c30 0x7fad2003b0f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fad1c009ad0 tx=0x7fad1c009210 comp rx=0 tx=0).stop 2023-07-20T01:22:37.103 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:37.121+0000 7fad39b0b700 1 -- 172.21.15.183:0/2780441124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fad340ff280 msgr2=0x7fad34195570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:37.103 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:37.121+0000 7fad39b0b700 1 --2- 172.21.15.183:0/2780441124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fad340ff280 0x7fad34195570 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fad24000c00 tx=0x7fad2400bcf0 comp rx=0 tx=0).stop 2023-07-20T01:22:37.103 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:37.121+0000 7fad39b0b700 1 -- 172.21.15.183:0/2780441124 shutdown_connections 2023-07-20T01:22:37.104 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:37.121+0000 7fad39b0b700 1 --2- 172.21.15.183:0/2780441124 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fad20038c30 0x7fad2003b0f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:37.104 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:37.121+0000 7fad39b0b700 1 --2- 172.21.15.183:0/2780441124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fad34195f30 0x7fad3419d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:37.104 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:37.121+0000 7fad39b0b700 1 --2- 172.21.15.183:0/2780441124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fad340ff280 0x7fad34195570 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:37.104 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:37.121+0000 7fad39b0b700 1 --2- 172.21.15.183:0/2780441124 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad34195ab0 0x7fad3419ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:37.104 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:37.122+0000 7fad39b0b700 1 -- 172.21.15.183:0/2780441124 >> 172.21.15.183:0/2780441124 conn(0x7fad340fa840 msgr2=0x7fad34068760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:22:37.104 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:37.122+0000 7fad39b0b700 1 -- 172.21.15.183:0/2780441124 shutdown_connections 2023-07-20T01:22:37.104 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:22:37.122+0000 7fad39b0b700 1 -- 172.21.15.183:0/2780441124 wait complete. 2023-07-20T01:22:37.124 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:36 smithi183 ceph-mon[98477]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:37.125 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:36 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:37.125 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:36 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:37.125 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:36 smithi183 ceph-mon[98477]: Reconfiguring mon.c (monmap changed)... 2023-07-20T01:22:37.125 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:36 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T01:22:37.125 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:36 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T01:22:37.125 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:36 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:37.125 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:36 smithi183 ceph-mon[98477]: Reconfiguring daemon mon.c on smithi183 2023-07-20T01:22:37.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:36 smithi136 ceph-mon[98083]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:37.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:36 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:37.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:36 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:37.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:36 smithi136 ceph-mon[98083]: Reconfiguring mon.c (monmap changed)... 2023-07-20T01:22:37.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:36 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T01:22:37.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:36 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T01:22:37.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:36 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:37.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:36 smithi136 ceph-mon[98083]: Reconfiguring daemon mon.c on smithi183 2023-07-20T01:22:37.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:36 smithi060 ceph-mon[96524]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:37.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:36 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:37.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:36 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:37.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:36 smithi060 ceph-mon[96524]: Reconfiguring mon.c (monmap changed)... 2023-07-20T01:22:37.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:36 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-20T01:22:37.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:36 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-20T01:22:37.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:36 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:37.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:36 smithi060 ceph-mon[96524]: Reconfiguring daemon mon.c on smithi183 2023-07-20T01:22:37.513 DEBUG:teuthology.orchestra.run.smithi136:mgr.y> sudo journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mgr.y.service 2023-07-20T01:22:37.516 DEBUG:teuthology.orchestra.run.smithi183:mgr.z> sudo journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mgr.z.service 2023-07-20T01:22:37.518 INFO:tasks.cephadm:Deploying OSDs... 2023-07-20T01:22:37.518 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2023-07-20T01:22:37.519 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/scratch_devs of=/dev/stdout 2023-07-20T01:22:37.540 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-20T01:22:37.540 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_1 2023-07-20T01:22:37.546 INFO:journalctl@ceph.mgr.z.smithi183.stdout:-- Logs begin at Thu 2023-07-20 01:00:08 UTC. -- 2023-07-20T01:22:37.550 INFO:journalctl@ceph.mgr.y.smithi136.stdout:-- Logs begin at Thu 2023-07-20 01:00:24 UTC. -- 2023-07-20T01:22:37.602 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-20T01:22:37.602 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T01:22:37.602 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 6h/6d Inode: 371510 Links: 1 2023-07-20T01:22:37.602 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T01:22:37.602 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T01:22:37.603 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2023-07-20 01:21:49.171008479 +0000 2023-07-20T01:22:37.603 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2023-07-20 01:10:45.149856144 +0000 2023-07-20T01:22:37.603 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2023-07-20 01:10:45.149856144 +0000 2023-07-20T01:22:37.603 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: - 2023-07-20T01:22:37.603 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-20T01:22:37.674 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2023-07-20T01:22:37.674 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2023-07-20T01:22:37.674 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.00012199 s, 4.2 MB/s 2023-07-20T01:22:37.675 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-20T01:22:37.735 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_2 2023-07-20T01:22:37.794 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-20T01:22:37.795 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T01:22:37.795 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 6h/6d Inode: 369542 Links: 1 2023-07-20T01:22:37.795 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T01:22:37.795 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T01:22:37.795 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2023-07-20 01:21:49.171008479 +0000 2023-07-20T01:22:37.795 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2023-07-20 01:10:45.661852027 +0000 2023-07-20T01:22:37.795 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2023-07-20 01:10:45.661852027 +0000 2023-07-20T01:22:37.796 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: - 2023-07-20T01:22:37.796 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-20T01:22:37.863 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2023-07-20T01:22:37.863 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2023-07-20T01:22:37.863 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000121838 s, 4.2 MB/s 2023-07-20T01:22:37.864 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-20T01:22:37.924 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_3 2023-07-20T01:22:37.982 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-20T01:22:37.983 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T01:22:37.983 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 6h/6d Inode: 377651 Links: 1 2023-07-20T01:22:37.983 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T01:22:37.983 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T01:22:37.983 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2023-07-20 01:21:49.171008479 +0000 2023-07-20T01:22:37.983 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2023-07-20 01:10:46.170847934 +0000 2023-07-20T01:22:37.983 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2023-07-20 01:10:46.170847934 +0000 2023-07-20T01:22:37.984 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: - 2023-07-20T01:22:37.984 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-20T01:22:38.050 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2023-07-20T01:22:38.050 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2023-07-20T01:22:38.050 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000113176 s, 4.5 MB/s 2023-07-20T01:22:38.051 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-20T01:22:38.110 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_4 2023-07-20T01:22:38.169 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-20T01:22:38.169 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T01:22:38.169 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 6h/6d Inode: 382992 Links: 1 2023-07-20T01:22:38.169 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T01:22:38.170 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T01:22:38.170 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2023-07-20 01:21:49.171008479 +0000 2023-07-20T01:22:38.170 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2023-07-20 01:10:46.720843512 +0000 2023-07-20T01:22:38.170 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2023-07-20 01:10:46.720843512 +0000 2023-07-20T01:22:38.170 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: - 2023-07-20T01:22:38.170 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-20T01:22:38.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:38 smithi183 ceph-mon[98477]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi060=x;smithi136=y;smithi183=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:22:38.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:38 smithi183 ceph-mon[98477]: Saving service mgr spec with placement smithi060=x;smithi136=y;smithi183=z;count:3 2023-07-20T01:22:38.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:38 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:38.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:38 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:38.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:38 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:38.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:38 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:22:38.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:38 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:38.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:38 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:22:38.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:38 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:38.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:38 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T01:22:38.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:38 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T01:22:38.203 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:38 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T01:22:38.203 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:38 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:38.203 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:38 smithi183 ceph-mon[98477]: Deploying daemon mgr.z on smithi183 2023-07-20T01:22:38.237 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2023-07-20T01:22:38.237 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2023-07-20T01:22:38.238 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000115655 s, 4.4 MB/s 2023-07-20T01:22:38.239 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-20T01:22:38.298 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-07-20T01:22:38.298 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/scratch_devs of=/dev/stdout 2023-07-20T01:22:38.319 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-20T01:22:38.319 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_1 2023-07-20T01:22:38.379 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-20T01:22:38.380 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T01:22:38.380 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 6h/6d Inode: 373399 Links: 1 2023-07-20T01:22:38.380 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T01:22:38.380 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T01:22:38.381 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2023-07-20 01:22:25.033235444 +0000 2023-07-20T01:22:38.381 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2023-07-20 01:10:01.882457586 +0000 2023-07-20T01:22:38.381 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2023-07-20 01:10:01.882457586 +0000 2023-07-20T01:22:38.381 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2023-07-20T01:22:38.381 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-20T01:22:38.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:38 smithi136 ceph-mon[98083]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi060=x;smithi136=y;smithi183=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:22:38.399 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:38 smithi136 ceph-mon[98083]: Saving service mgr spec with placement smithi060=x;smithi136=y;smithi183=z;count:3 2023-07-20T01:22:38.399 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:38 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:38.399 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:38 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:38.399 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:38 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:38.399 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:38 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:22:38.399 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:38 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:38.399 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:38 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:22:38.399 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:38 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:38.400 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:38 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T01:22:38.400 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:38 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T01:22:38.400 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:38 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T01:22:38.400 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:38 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:38.400 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:38 smithi136 ceph-mon[98083]: Deploying daemon mgr.z on smithi183 2023-07-20T01:22:38.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:38 smithi060 ceph-mon[96524]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi060=x;smithi136=y;smithi183=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:22:38.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:38 smithi060 ceph-mon[96524]: Saving service mgr spec with placement smithi060=x;smithi136=y;smithi183=z;count:3 2023-07-20T01:22:38.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:38 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:38.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:38 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:38.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:38 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:38.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:38 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:22:38.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:38 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:38.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:38 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:22:38.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:38 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:38.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:38 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T01:22:38.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:38 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T01:22:38.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:38 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T01:22:38.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:38 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:38.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:38 smithi060 ceph-mon[96524]: Deploying daemon mgr.z on smithi183 2023-07-20T01:22:38.430 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2023-07-20T01:22:38.431 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2023-07-20T01:22:38.431 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000270999 s, 1.9 MB/s 2023-07-20T01:22:38.431 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-20T01:22:38.490 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_2 2023-07-20T01:22:38.550 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-20T01:22:38.550 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T01:22:38.550 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 6h/6d Inode: 379859 Links: 1 2023-07-20T01:22:38.550 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T01:22:38.550 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T01:22:38.551 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2023-07-20 01:22:25.033235444 +0000 2023-07-20T01:22:38.551 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2023-07-20 01:10:02.402446262 +0000 2023-07-20T01:22:38.551 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2023-07-20 01:10:02.402446262 +0000 2023-07-20T01:22:38.551 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2023-07-20T01:22:38.551 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-20T01:22:38.617 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2023-07-20T01:22:38.618 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2023-07-20T01:22:38.618 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000233183 s, 2.2 MB/s 2023-07-20T01:22:38.619 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-20T01:22:38.678 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_3 2023-07-20T01:22:38.736 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-20T01:22:38.737 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T01:22:38.737 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 6h/6d Inode: 377693 Links: 1 2023-07-20T01:22:38.737 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T01:22:38.737 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T01:22:38.737 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2023-07-20 01:22:25.033235444 +0000 2023-07-20T01:22:38.737 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2023-07-20 01:10:02.919435002 +0000 2023-07-20T01:22:38.738 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2023-07-20 01:10:02.919435002 +0000 2023-07-20T01:22:38.738 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2023-07-20T01:22:38.738 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-20T01:22:38.803 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2023-07-20T01:22:38.804 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2023-07-20T01:22:38.804 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000226731 s, 2.3 MB/s 2023-07-20T01:22:38.805 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-20T01:22:38.863 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_4 2023-07-20T01:22:38.920 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-20T01:22:38.920 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T01:22:38.920 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 6h/6d Inode: 380221 Links: 1 2023-07-20T01:22:38.921 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T01:22:38.921 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T01:22:38.921 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2023-07-20 01:22:25.033235444 +0000 2023-07-20T01:22:38.921 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2023-07-20 01:10:03.455423329 +0000 2023-07-20T01:22:38.921 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2023-07-20 01:10:03.455423329 +0000 2023-07-20T01:22:38.921 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2023-07-20T01:22:38.922 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-20T01:22:38.986 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2023-07-20T01:22:38.986 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2023-07-20T01:22:38.986 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000227942 s, 2.2 MB/s 2023-07-20T01:22:38.987 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-20T01:22:39.045 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2023-07-20T01:22:39.045 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/scratch_devs of=/dev/stdout 2023-07-20T01:22:39.064 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-20T01:22:39.065 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_1 2023-07-20T01:22:39.121 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-20T01:22:39.122 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T01:22:39.122 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 6h/6d Inode: 372424 Links: 1 2023-07-20T01:22:39.122 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T01:22:39.122 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T01:22:39.123 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2023-07-20 01:22:38.823588384 +0000 2023-07-20T01:22:39.123 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2023-07-20 01:10:15.104982134 +0000 2023-07-20T01:22:39.123 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2023-07-20 01:10:15.104982134 +0000 2023-07-20T01:22:39.123 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: - 2023-07-20T01:22:39.123 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-20T01:22:39.143 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:22:38 smithi183 systemd[1]: Starting Ceph mgr.z for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:22:39.172 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2023-07-20T01:22:39.172 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2023-07-20T01:22:39.172 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.00024842 s, 2.1 MB/s 2023-07-20T01:22:39.173 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-20T01:22:39.231 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_2 2023-07-20T01:22:39.289 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-20T01:22:39.289 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T01:22:39.289 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 6h/6d Inode: 372489 Links: 1 2023-07-20T01:22:39.290 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T01:22:39.290 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T01:22:39.290 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2023-07-20 01:22:38.823588384 +0000 2023-07-20T01:22:39.290 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2023-07-20 01:10:15.612969012 +0000 2023-07-20T01:22:39.290 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2023-07-20 01:10:15.612969012 +0000 2023-07-20T01:22:39.290 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: - 2023-07-20T01:22:39.291 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-20T01:22:39.355 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2023-07-20T01:22:39.355 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2023-07-20T01:22:39.355 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000227906 s, 2.2 MB/s 2023-07-20T01:22:39.356 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-20T01:22:39.414 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_3 2023-07-20T01:22:39.415 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:39 smithi183 ceph-mon[98477]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:39.415 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:22:39 smithi183 podman[100325]: 2023-07-20T01:22:39.431 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-20T01:22:39.431 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T01:22:39.432 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 6h/6d Inode: 372569 Links: 1 2023-07-20T01:22:39.432 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T01:22:39.432 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T01:22:39.432 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2023-07-20 01:22:38.823588384 +0000 2023-07-20T01:22:39.432 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2023-07-20 01:10:16.128955683 +0000 2023-07-20T01:22:39.432 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2023-07-20 01:10:16.128955683 +0000 2023-07-20T01:22:39.432 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: - 2023-07-20T01:22:39.433 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-20T01:22:39.500 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2023-07-20T01:22:39.500 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2023-07-20T01:22:39.500 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000235712 s, 2.2 MB/s 2023-07-20T01:22:39.501 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-20T01:22:39.558 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_4 2023-07-20T01:22:39.614 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-20T01:22:39.615 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-20T01:22:39.615 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 6h/6d Inode: 380613 Links: 1 2023-07-20T01:22:39.615 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-20T01:22:39.615 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2023-07-20T01:22:39.615 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2023-07-20 01:22:38.823588384 +0000 2023-07-20T01:22:39.615 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2023-07-20 01:10:16.640942458 +0000 2023-07-20T01:22:39.616 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2023-07-20 01:10:16.640942458 +0000 2023-07-20T01:22:39.616 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: - 2023-07-20T01:22:39.616 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-20T01:22:39.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:39 smithi136 ceph-mon[98083]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:39.677 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:22:39 smithi183 bash[100325]: 3a5bf8e7e65dcdbae9462e034188f593d6642f93df7c65c858a9bc7ad3ff3cf1 2023-07-20T01:22:39.678 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:22:39 smithi183 ceph-mgr[100486]: -- 172.21.15.183:0/3465213304 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56503ed911e0 con 0x56503ed67000 2023-07-20T01:22:39.678 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:22:39 smithi183 systemd[1]: Started Ceph mgr.z for 7daf031e-269b-11ee-9b34-001a4aab830c. 2023-07-20T01:22:39.678 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:22:39 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-z[100399]: 2023-07-20T01:22:39.559+0000 7f52ae38b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T01:22:39.678 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:22:39 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-z[100399]: 2023-07-20T01:22:39.642+0000 7f52ae38b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T01:22:39.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:39 smithi060 ceph-mon[96524]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:39.680 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2023-07-20T01:22:39.680 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2023-07-20T01:22:39.680 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000186845 s, 2.7 MB/s 2023-07-20T01:22:39.681 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-20T01:22:39.738 INFO:tasks.cephadm:Deploying osd.0 on smithi060 with /dev/vg_nvme/lv_4... 2023-07-20T01:22:39.739 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-20T01:22:39.951 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:22:39 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-z[100399]: 2023-07-20T01:22:39.735+0000 7f52ae38b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T01:22:39.951 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:22:39 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-z[100399]: 2023-07-20T01:22:39.832+0000 7f52ae38b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T01:22:40.286 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:22:40 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-z[100399]: 2023-07-20T01:22:40.000+0000 7f52ae38b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T01:22:40.310 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:22:40.595 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:22:40 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-z[100399]: 2023-07-20T01:22:40.304+0000 7f52ae38b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T01:22:40.596 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:22:40 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-z[100399]: 2023-07-20T01:22:40.392+0000 7f52ae38b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T01:22:40.596 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:22:40 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-z[100399]: 2023-07-20T01:22:40.520+0000 7f52ae38b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T01:22:40.596 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:40 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:40.596 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:40 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:40.597 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:40 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:40.597 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:40 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T01:22:40.597 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:40 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T01:22:40.597 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:40 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T01:22:40.597 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:40 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:40.597 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:40 smithi183 ceph-mon[98477]: Deploying daemon mgr.y on smithi136 2023-07-20T01:22:40.772 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:40 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:40.772 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:40 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:40.772 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:40 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:40.772 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:40 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T01:22:40.772 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:40 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T01:22:40.772 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:40 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T01:22:40.773 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:40 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:40.773 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:40 smithi136 ceph-mon[98083]: Deploying daemon mgr.y on smithi136 2023-07-20T01:22:40.802 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:40 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:40.802 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:40 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:40.802 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:40 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:40.802 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:40 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T01:22:40.802 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:40 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-20T01:22:40.803 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:40 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T01:22:40.803 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:40 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:40.803 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:40 smithi060 ceph-mon[96524]: Deploying daemon mgr.y on smithi136 2023-07-20T01:22:40.951 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:22:40 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-z[100399]: 2023-07-20T01:22:40.614+0000 7f52ae38b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T01:22:41.398 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:22:41 smithi136 systemd[1]: Starting Ceph mgr.y for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:22:41.451 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:22:41 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-z[100399]: 2023-07-20T01:22:41.032+0000 7f52ae38b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T01:22:41.670 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:41 smithi136 ceph-mon[98083]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:41.670 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:22:41 smithi136 podman[99522]: 2023-07-20T01:22:41.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:41 smithi060 ceph-mon[96524]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:41.860 INFO:teuthology.orchestra.run.smithi060.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-20T01:22:41.860 INFO:teuthology.orchestra.run.smithi060.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-20T01:22:41.860 INFO:teuthology.orchestra.run.smithi060.stderr: stderr: 10+0 records in 2023-07-20T01:22:41.860 INFO:teuthology.orchestra.run.smithi060.stderr:10+0 records out 2023-07-20T01:22:41.860 INFO:teuthology.orchestra.run.smithi060.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00810016 s, 1.3 GB/s 2023-07-20T01:22:41.861 INFO:teuthology.orchestra.run.smithi060.stderr: stderr: 2023-07-20T01:22:41.861 INFO:teuthology.orchestra.run.smithi060.stderr:--> Zapping successful for: 2023-07-20T01:22:41.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:41 smithi183 ceph-mon[98477]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:41.951 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:22:41 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-z[100399]: 2023-07-20T01:22:41.604+0000 7f52ae38b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T01:22:41.951 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:22:41 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-z[100399]: 2023-07-20T01:22:41.774+0000 7f52ae38b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T01:22:41.963 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:22:41 smithi136 ceph-mgr[99562]: -- 172.21.15.136:0/3664307991 <== mon.2 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ffe1ab71e0 con 0x55ffe1a8cc00 2023-07-20T01:22:41.963 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:22:41 smithi136 bash[99522]: 93c08c3e7e71f0f90082a9bb6da1acce404e4cca0fb2f4ac038e8cc0a5d28cce 2023-07-20T01:22:41.963 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:22:41 smithi136 systemd[1]: Started Ceph mgr.y for 7daf031e-269b-11ee-9b34-001a4aab830c. 2023-07-20T01:22:41.963 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:22:41 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-y[99538]: 2023-07-20T01:22:41.810+0000 7f8a70a8e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-20T01:22:41.963 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:22:41 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-y[99538]: 2023-07-20T01:22:41.890+0000 7f8a70a8e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-20T01:22:41.964 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:22:41 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-y[99538]: 2023-07-20T01:22:41.981+0000 7f8a70a8e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-20T01:22:42.245 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:22:42 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-y[99538]: 2023-07-20T01:22:42.078+0000 7f8a70a8e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-20T01:22:42.268 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi060:vg_nvme/lv_4 2023-07-20T01:22:42.562 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:22:42 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-y[99538]: 2023-07-20T01:22:42.263+0000 7f8a70a8e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-20T01:22:42.828 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:22:42.843 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:22:42 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-z[100399]: 2023-07-20T01:22:42.589+0000 7f52ae38b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T01:22:42.843 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:22:42 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-z[100399]: 2023-07-20T01:22:42.685+0000 7f52ae38b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T01:22:42.844 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:42 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:42.844 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:42 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:42.844 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:42 smithi183 ceph-mon[98477]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:42.844 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:42 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:42.845 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:42 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:42.845 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:42 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:22:42.886 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:22:42 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-y[99538]: 2023-07-20T01:22:42.580+0000 7f8a70a8e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-20T01:22:42.887 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:22:42 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-y[99538]: 2023-07-20T01:22:42.671+0000 7f8a70a8e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-20T01:22:42.887 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:22:42 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-y[99538]: 2023-07-20T01:22:42.804+0000 7f8a70a8e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-20T01:22:42.888 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:42 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:42.888 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:42 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:42.889 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:42 smithi136 ceph-mon[98083]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:42.889 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:42 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:42.889 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:42 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:42.889 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:42 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:22:43.045 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:42 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:43.045 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:42 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:43.045 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:42 smithi060 ceph-mon[96524]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:43.045 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:42 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:43.045 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:42 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:43.046 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:42 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:22:43.131 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:22:42 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-z[100399]: 2023-07-20T01:22:42.862+0000 7f52ae38b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T01:22:43.148 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:22:42 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-y[99538]: 2023-07-20T01:22:42.904+0000 7f8a70a8e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-20T01:22:43.451 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:22:43 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-z[100399]: 2023-07-20T01:22:43.150+0000 7f52ae38b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T01:22:43.451 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:22:43 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-z[100399]: 2023-07-20T01:22:43.390+0000 7f52ae38b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T01:22:43.648 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:22:43 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-y[99538]: 2023-07-20T01:22:43.352+0000 7f8a70a8e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-20T01:22:43.792 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:22:43 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-z[100399]: 2023-07-20T01:22:43.546+0000 7f52ae38b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T01:22:43.793 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:22:43 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-z[100399]: 2023-07-20T01:22:43.811+0000 7f52ae38b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T01:22:43.893 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:43 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:43.893 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:43 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:43.894 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:43 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:43.894 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:43 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:43.894 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:43 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:43.894 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:43 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:43.894 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:43 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:22:43.894 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:43 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:43.894 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:43 smithi060 ceph-mon[96524]: Reconfiguring mgr.x (unknown last config time)... 2023-07-20T01:22:43.895 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:43 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T01:22:43.895 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:43 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T01:22:43.895 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:43 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:43.895 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:43 smithi060 ceph-mon[96524]: Reconfiguring daemon mgr.x on smithi060 2023-07-20T01:22:44.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.116+0000 7faac2f82700 1 -- 172.21.15.60:0/500562822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faabc100900 msgr2=0x7faabc100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:44.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.116+0000 7faac2f82700 1 --2- 172.21.15.60:0/500562822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faabc100900 0x7faabc100d60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7faaa80099c0 tx=0x7faaa8009cd0 comp rx=0 tx=0).stop 2023-07-20T01:22:44.098 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.116+0000 7faac2f82700 1 -- 172.21.15.60:0/500562822 shutdown_connections 2023-07-20T01:22:44.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.116+0000 7faac2f82700 1 --2- 172.21.15.60:0/500562822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faabc1012a0 0x7faabc107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:44.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.116+0000 7faac2f82700 1 --2- 172.21.15.60:0/500562822 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faabc0ff700 0x7faabc0ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:44.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.116+0000 7faac2f82700 1 --2- 172.21.15.60:0/500562822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faabc100900 0x7faabc100d60 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:44.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.116+0000 7faac2f82700 1 -- 172.21.15.60:0/500562822 >> 172.21.15.60:0/500562822 conn(0x7faabc0fac80 msgr2=0x7faabc0fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:22:44.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.116+0000 7faac2f82700 1 -- 172.21.15.60:0/500562822 shutdown_connections 2023-07-20T01:22:44.099 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.116+0000 7faac2f82700 1 -- 172.21.15.60:0/500562822 wait complete. 2023-07-20T01:22:44.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.117+0000 7faac2f82700 1 Processor -- start 2023-07-20T01:22:44.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.117+0000 7faac2f82700 1 -- start start 2023-07-20T01:22:44.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.117+0000 7faac2f82700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faabc0ff700 0x7faabc068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:22:44.101 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.117+0000 7faac2f82700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faabc100900 0x7faabc19b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:22:44.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.117+0000 7faac2f82700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faabc1012a0 0x7faabc19d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:22:44.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.117+0000 7faac2f82700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faabc19df60 con 0x7faabc1012a0 2023-07-20T01:22:44.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.117+0000 7faac2f82700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faabc19e0d0 con 0x7faabc100900 2023-07-20T01:22:44.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.117+0000 7faac2f82700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faabc19e290 con 0x7faabc0ff700 2023-07-20T01:22:44.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.117+0000 7faac151f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faabc1012a0 0x7faabc19d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:22:44.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.117+0000 7faac151f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faabc1012a0 0x7faabc19d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42224/0 (socket says 172.21.15.60:42224) 2023-07-20T01:22:44.102 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.117+0000 7faac151f700 1 -- 172.21.15.60:0/1755977111 learned_addr learned my addr 172.21.15.60:0/1755977111 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:22:44.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.117+0000 7faac0d1e700 1 --2- 172.21.15.60:0/1755977111 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faabc0ff700 0x7faabc068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:22:44.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.117+0000 7faabbfff700 1 --2- 172.21.15.60:0/1755977111 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faabc100900 0x7faabc19b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:22:44.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.118+0000 7faac151f700 1 -- 172.21.15.60:0/1755977111 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faabc0ff700 msgr2=0x7faabc068d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:44.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.118+0000 7faac151f700 1 --2- 172.21.15.60:0/1755977111 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faabc0ff700 0x7faabc068d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:44.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.118+0000 7faac151f700 1 -- 172.21.15.60:0/1755977111 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faabc100900 msgr2=0x7faabc19b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:22:44.103 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.118+0000 7faac151f700 1 --2- 172.21.15.60:0/1755977111 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faabc100900 0x7faabc19b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:22:44.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.118+0000 7faac151f700 1 -- 172.21.15.60:0/1755977111 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa80095f0 con 0x7faabc1012a0 2023-07-20T01:22:44.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.118+0000 7faac151f700 1 --2- 172.21.15.60:0/1755977111 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faabc1012a0 0x7faabc19d7c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7faab4007ee0 tx=0x7faab400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:22:44.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.118+0000 7faab9ffb700 1 -- 172.21.15.60:0/1755977111 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab4017070 con 0x7faabc1012a0 2023-07-20T01:22:44.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.118+0000 7faab9ffb700 1 -- 172.21.15.60:0/1755977111 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7faab400f040 con 0x7faabc1012a0 2023-07-20T01:22:44.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.118+0000 7faac2f82700 1 -- 172.21.15.60:0/1755977111 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faabc19e570 con 0x7faabc1012a0 2023-07-20T01:22:44.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.118+0000 7faab9ffb700 1 -- 172.21.15.60:0/1755977111 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab4005570 con 0x7faabc1012a0 2023-07-20T01:22:44.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.118+0000 7faac2f82700 1 -- 172.21.15.60:0/1755977111 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faabc19ea40 con 0x7faabc1012a0 2023-07-20T01:22:44.104 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.119+0000 7faab9ffb700 1 -- 172.21.15.60:0/1755977111 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7faab400db20 con 0x7faabc1012a0 2023-07-20T01:22:44.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.119+0000 7faac2f82700 1 -- 172.21.15.60:0/1755977111 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faabc1066f0 con 0x7faabc1012a0 2023-07-20T01:22:44.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.119+0000 7faab9ffb700 1 --2- 172.21.15.60:0/1755977111 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7faaac0389e0 0x7faaac03aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:22:44.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.119+0000 7faab9ffb700 1 -- 172.21.15.60:0/1755977111 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7faab404aa20 con 0x7faabc1012a0 2023-07-20T01:22:44.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.120+0000 7faac0d1e700 1 --2- 172.21.15.60:0/1755977111 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7faaac0389e0 0x7faaac03aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:22:44.105 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.120+0000 7faac0d1e700 1 --2- 172.21.15.60:0/1755977111 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7faaac0389e0 0x7faaac03aea0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7faabc100760 tx=0x7faab0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:22:44.106 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.124+0000 7faab9ffb700 1 -- 172.21.15.60:0/1755977111 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7faab4004c40 con 0x7faabc1012a0 2023-07-20T01:22:44.148 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:22:43 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-y[99538]: 2023-07-20T01:22:43.944+0000 7f8a70a8e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-20T01:22:44.148 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:22:44 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-y[99538]: 2023-07-20T01:22:44.114+0000 7f8a70a8e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-20T01:22:44.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:43 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:44.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:43 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:44.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:43 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:44.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:43 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:44.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:43 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:44.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:43 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:44.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:43 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:22:44.150 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:43 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:44.150 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:43 smithi136 ceph-mon[98083]: Reconfiguring mgr.x (unknown last config time)... 2023-07-20T01:22:44.150 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:43 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T01:22:44.150 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:43 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T01:22:44.150 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:43 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:44.150 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:43 smithi136 ceph-mon[98083]: Reconfiguring daemon mgr.x on smithi060 2023-07-20T01:22:44.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:43 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:44.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:43 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:44.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:43 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:44.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:43 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:44.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:43 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:44.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:43 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:44.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:43 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:22:44.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:43 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:44.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:43 smithi183 ceph-mon[98477]: Reconfiguring mgr.x (unknown last config time)... 2023-07-20T01:22:44.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:43 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-20T01:22:44.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:43 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-20T01:22:44.203 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:43 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:44.203 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:43 smithi183 ceph-mon[98477]: Reconfiguring daemon mgr.x on smithi060 2023-07-20T01:22:44.315 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:44.334+0000 7faac2f82700 1 -- 172.21.15.60:0/1755977111 --> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7faabc060bb0 con 0x7faaac0389e0 2023-07-20T01:22:44.817 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:22:44 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-z[100399]: 2023-07-20T01:22:44.514+0000 7f52ae38b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T01:22:44.818 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:22:44 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-z[100399]: 2023-07-20T01:22:44.645+0000 7f52ae38b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T01:22:44.818 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:22:44 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-z[100399]: 2023-07-20T01:22:44.836+0000 7f52ae38b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T01:22:45.068 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:44 smithi060 ceph-mon[96524]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:45.068 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:44 smithi060 ceph-mon[96524]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:22:45.069 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:44 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:22:45.069 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:44 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:22:45.069 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:44 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:45.121 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:22:44 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-y[99538]: 2023-07-20T01:22:44.878+0000 7f8a70a8e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-20T01:22:45.121 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:22:44 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-y[99538]: 2023-07-20T01:22:44.968+0000 7f8a70a8e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-20T01:22:45.121 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:44 smithi136 ceph-mon[98083]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:45.121 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:44 smithi136 ceph-mon[98083]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:22:45.121 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:44 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:22:45.122 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:44 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:22:45.122 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:44 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:45.149 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:22:44 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-z[100399]: 2023-07-20T01:22:44.950+0000 7f52ae38b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T01:22:45.150 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:22:45 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-z[100399]: 2023-07-20T01:22:45.168+0000 7f52ae38b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T01:22:45.150 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:44 smithi183 ceph-mon[98477]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:45.150 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:44 smithi183 ceph-mon[98477]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:22:45.150 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:44 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:22:45.150 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:44 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:22:45.150 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:44 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:45.398 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:22:45 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-y[99538]: 2023-07-20T01:22:45.139+0000 7f8a70a8e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-20T01:22:45.774 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:22:45 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-y[99538]: 2023-07-20T01:22:45.418+0000 7f8a70a8e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-20T01:22:45.774 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:22:45 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-y[99538]: 2023-07-20T01:22:45.647+0000 7f8a70a8e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-20T01:22:46.030 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:22:45 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-y[99538]: 2023-07-20T01:22:45.792+0000 7f8a70a8e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-20T01:22:46.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:46.148+0000 7faab9ffb700 1 -- 172.21.15.60:0/1755977111 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 14) v1 ==== 90599+0+0 (secure 0 0 0) 0x7faab401c790 con 0x7faabc1012a0 2023-07-20T01:22:46.398 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:22:46 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-y[99538]: 2023-07-20T01:22:46.048+0000 7f8a70a8e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-20T01:22:46.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:46 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:46.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:46 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:46.399 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:46 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:22:46.399 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:46 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:46.399 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:46 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:22:46.399 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:46 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:46.399 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:46 smithi136 ceph-mon[98083]: Standby manager daemon z started 2023-07-20T01:22:46.399 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:46 smithi136 ceph-mon[98083]: from='mgr.? 172.21.15.183:0/4195647183' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-20T01:22:46.399 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:46 smithi136 ceph-mon[98083]: from='mgr.? 172.21.15.183:0/4195647183' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T01:22:46.400 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:46 smithi136 ceph-mon[98083]: from='mgr.? 172.21.15.183:0/4195647183' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-20T01:22:46.400 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:46 smithi136 ceph-mon[98083]: from='mgr.? 172.21.15.183:0/4195647183' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T01:22:46.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:46 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:46.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:46 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:46.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:46 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:22:46.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:46 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:46.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:46 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:22:46.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:46 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:46.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:46 smithi060 ceph-mon[96524]: Standby manager daemon z started 2023-07-20T01:22:46.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:46 smithi060 ceph-mon[96524]: from='mgr.? 172.21.15.183:0/4195647183' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-20T01:22:46.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:46 smithi060 ceph-mon[96524]: from='mgr.? 172.21.15.183:0/4195647183' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T01:22:46.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:46 smithi060 ceph-mon[96524]: from='mgr.? 172.21.15.183:0/4195647183' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-20T01:22:46.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:46 smithi060 ceph-mon[96524]: from='mgr.? 172.21.15.183:0/4195647183' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T01:22:46.451 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:22:46 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-z[100399]: 2023-07-20T01:22:46.028+0000 7f52ae38b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T01:22:46.451 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:46 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:46.451 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:46 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:46.451 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:46 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:22:46.452 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:46 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:46.452 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:46 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:22:46.452 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:46 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:46.452 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:46 smithi183 ceph-mon[98477]: Standby manager daemon z started 2023-07-20T01:22:46.452 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:46 smithi183 ceph-mon[98477]: from='mgr.? 172.21.15.183:0/4195647183' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-20T01:22:46.452 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:46 smithi183 ceph-mon[98477]: from='mgr.? 172.21.15.183:0/4195647183' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T01:22:46.453 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:46 smithi183 ceph-mon[98477]: from='mgr.? 172.21.15.183:0/4195647183' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-20T01:22:46.453 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:46 smithi183 ceph-mon[98477]: from='mgr.? 172.21.15.183:0/4195647183' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T01:22:47.051 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:22:46 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-y[99538]: 2023-07-20T01:22:46.747+0000 7f8a70a8e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-20T01:22:47.051 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:22:46 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-y[99538]: 2023-07-20T01:22:46.878+0000 7f8a70a8e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-20T01:22:47.378 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:22:47 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-y[99538]: 2023-07-20T01:22:47.069+0000 7f8a70a8e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-20T01:22:47.379 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:22:47 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-y[99538]: 2023-07-20T01:22:47.182+0000 7f8a70a8e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-20T01:22:47.379 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:47 smithi136 ceph-mon[98083]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:47.379 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:47 smithi136 ceph-mon[98083]: mgrmap e14: x(active, since 68s), standbys: z 2023-07-20T01:22:47.379 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:47 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-20T01:22:47.379 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:47 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/3305771008' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9e5d1e7-1bfe-445f-8907-a1bf68b56c46"}]: dispatch 2023-07-20T01:22:47.380 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:47 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/3305771008' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c9e5d1e7-1bfe-445f-8907-a1bf68b56c46"}]': finished 2023-07-20T01:22:47.380 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:47 smithi136 ceph-mon[98083]: osdmap e5: 1 total, 0 up, 1 in 2023-07-20T01:22:47.380 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:47 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:22:47.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:47 smithi060 ceph-mon[96524]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:47.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:47 smithi060 ceph-mon[96524]: mgrmap e14: x(active, since 68s), standbys: z 2023-07-20T01:22:47.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:47 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-20T01:22:47.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:47 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/3305771008' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9e5d1e7-1bfe-445f-8907-a1bf68b56c46"}]: dispatch 2023-07-20T01:22:47.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:47 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/3305771008' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c9e5d1e7-1bfe-445f-8907-a1bf68b56c46"}]': finished 2023-07-20T01:22:47.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:47 smithi060 ceph-mon[96524]: osdmap e5: 1 total, 0 up, 1 in 2023-07-20T01:22:47.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:47 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:22:47.451 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:47 smithi183 ceph-mon[98477]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:47.451 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:47 smithi183 ceph-mon[98477]: mgrmap e14: x(active, since 68s), standbys: z 2023-07-20T01:22:47.451 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:47 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-20T01:22:47.451 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:47 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/3305771008' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9e5d1e7-1bfe-445f-8907-a1bf68b56c46"}]: dispatch 2023-07-20T01:22:47.451 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:47 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/3305771008' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c9e5d1e7-1bfe-445f-8907-a1bf68b56c46"}]': finished 2023-07-20T01:22:47.451 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:47 smithi183 ceph-mon[98477]: osdmap e5: 1 total, 0 up, 1 in 2023-07-20T01:22:47.452 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:47 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:22:47.648 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:22:47 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-y[99538]: 2023-07-20T01:22:47.397+0000 7f8a70a8e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-20T01:22:48.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:48 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/366767608' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:22:48.398 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:22:48 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-y[99538]: 2023-07-20T01:22:48.250+0000 7f8a70a8e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-20T01:22:48.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:48 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/366767608' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:22:48.451 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:48 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/366767608' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:22:49.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:22:49.164+0000 7faab9ffb700 1 -- 172.21.15.60:0/1755977111 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7faab40171d0 con 0x7faabc1012a0 2023-07-20T01:22:49.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:49 smithi136 ceph-mon[98083]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:49.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:49 smithi136 ceph-mon[98083]: Standby manager daemon y started 2023-07-20T01:22:49.399 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:49 smithi136 ceph-mon[98083]: from='mgr.? 172.21.15.136:0/3289872422' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-20T01:22:49.399 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:49 smithi136 ceph-mon[98083]: from='mgr.? 172.21.15.136:0/3289872422' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T01:22:49.399 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:49 smithi136 ceph-mon[98083]: from='mgr.? 172.21.15.136:0/3289872422' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-20T01:22:49.399 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:49 smithi136 ceph-mon[98083]: from='mgr.? 172.21.15.136:0/3289872422' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T01:22:49.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:49 smithi060 ceph-mon[96524]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:49.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:49 smithi060 ceph-mon[96524]: Standby manager daemon y started 2023-07-20T01:22:49.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:49 smithi060 ceph-mon[96524]: from='mgr.? 172.21.15.136:0/3289872422' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-20T01:22:49.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:49 smithi060 ceph-mon[96524]: from='mgr.? 172.21.15.136:0/3289872422' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T01:22:49.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:49 smithi060 ceph-mon[96524]: from='mgr.? 172.21.15.136:0/3289872422' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-20T01:22:49.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:49 smithi060 ceph-mon[96524]: from='mgr.? 172.21.15.136:0/3289872422' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T01:22:49.451 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:49 smithi183 ceph-mon[98477]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:49.451 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:49 smithi183 ceph-mon[98477]: Standby manager daemon y started 2023-07-20T01:22:49.452 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:49 smithi183 ceph-mon[98477]: from='mgr.? 172.21.15.136:0/3289872422' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-20T01:22:49.452 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:49 smithi183 ceph-mon[98477]: from='mgr.? 172.21.15.136:0/3289872422' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-20T01:22:49.452 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:49 smithi183 ceph-mon[98477]: from='mgr.? 172.21.15.136:0/3289872422' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-20T01:22:49.452 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:49 smithi183 ceph-mon[98477]: from='mgr.? 172.21.15.136:0/3289872422' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-20T01:22:50.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:50 smithi136 ceph-mon[98083]: mgrmap e15: x(active, since 71s), standbys: z, y 2023-07-20T01:22:50.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:50 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-20T01:22:50.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:50 smithi060 ceph-mon[96524]: mgrmap e15: x(active, since 71s), standbys: z, y 2023-07-20T01:22:50.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:50 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-20T01:22:50.451 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:50 smithi183 ceph-mon[98477]: mgrmap e15: x(active, since 71s), standbys: z, y 2023-07-20T01:22:50.451 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:50 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-20T01:22:51.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:51 smithi136 ceph-mon[98083]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:51.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:51 smithi060 ceph-mon[96524]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:51.451 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:51 smithi183 ceph-mon[98477]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:53.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:53 smithi060 ceph-mon[96524]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:53.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:53 smithi136 ceph-mon[98083]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:53.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:53 smithi183 ceph-mon[98477]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:54.466 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:54 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-20T01:22:54.466 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:54 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:54.467 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:54 smithi060 ceph-mon[96524]: Deploying daemon osd.0 on smithi060 2023-07-20T01:22:54.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:54 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-20T01:22:54.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:54 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:54.649 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:54 smithi136 ceph-mon[98083]: Deploying daemon osd.0 on smithi060 2023-07-20T01:22:54.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:54 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-20T01:22:54.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:54 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:22:54.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:54 smithi183 ceph-mon[98477]: Deploying daemon osd.0 on smithi060 2023-07-20T01:22:55.607 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:55 smithi060 ceph-mon[96524]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:55.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:55 smithi136 ceph-mon[98083]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:55.702 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:55 smithi183 ceph-mon[98477]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:57.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:57 smithi060 ceph-mon[96524]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:57.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:57 smithi136 ceph-mon[98083]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:57.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:57 smithi183 ceph-mon[98477]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:59.316 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:59 smithi060 ceph-mon[96524]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:59.316 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:59 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:59.316 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:59 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:59.316 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:22:59 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:22:59.451 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:59 smithi183 ceph-mon[98477]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:59.451 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:59 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:59.451 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:59 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:59.451 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:22:59 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:22:59.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:59 smithi136 ceph-mon[98083]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:22:59.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:59 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:59.649 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:59 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:22:59.649 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:22:59 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:23:01.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:01.087+0000 7faab9ffb700 1 -- 172.21.15.60:0/1755977111 <== mgr.14152 v2:172.21.15.60:6800/1961833180 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7faabc060bb0 con 0x7faaac0389e0 2023-07-20T01:23:01.069 INFO:teuthology.orchestra.run.smithi060.stdout:Created osd(s) 0 on host 'smithi060' 2023-07-20T01:23:01.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:01.090+0000 7faac2f82700 1 -- 172.21.15.60:0/1755977111 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7faaac0389e0 msgr2=0x7faaac03aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:23:01.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:01.090+0000 7faac2f82700 1 --2- 172.21.15.60:0/1755977111 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7faaac0389e0 0x7faaac03aea0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7faabc100760 tx=0x7faab0009260 comp rx=0 tx=0).stop 2023-07-20T01:23:01.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:01.090+0000 7faac2f82700 1 -- 172.21.15.60:0/1755977111 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faabc1012a0 msgr2=0x7faabc19d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:23:01.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:01.090+0000 7faac2f82700 1 --2- 172.21.15.60:0/1755977111 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faabc1012a0 0x7faabc19d7c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7faab4007ee0 tx=0x7faab400d3b0 comp rx=0 tx=0).stop 2023-07-20T01:23:01.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:01.090+0000 7faac2f82700 1 -- 172.21.15.60:0/1755977111 shutdown_connections 2023-07-20T01:23:01.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:01.090+0000 7faac2f82700 1 --2- 172.21.15.60:0/1755977111 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7faaac0389e0 0x7faaac03aea0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:23:01.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:01.090+0000 7faac2f82700 1 --2- 172.21.15.60:0/1755977111 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faabc1012a0 0x7faabc19d7c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:23:01.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:01.090+0000 7faac2f82700 1 --2- 172.21.15.60:0/1755977111 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faabc0ff700 0x7faabc068d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:23:01.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:01.090+0000 7faac2f82700 1 --2- 172.21.15.60:0/1755977111 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faabc100900 0x7faabc19b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:23:01.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:01.090+0000 7faac2f82700 1 -- 172.21.15.60:0/1755977111 >> 172.21.15.60:0/1755977111 conn(0x7faabc0fac80 msgr2=0x7faabc0fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:23:01.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:01.091+0000 7faac2f82700 1 -- 172.21.15.60:0/1755977111 shutdown_connections 2023-07-20T01:23:01.074 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:01.091+0000 7faac2f82700 1 -- 172.21.15.60:0/1755977111 wait complete. 2023-07-20T01:23:01.525 DEBUG:teuthology.orchestra.run.smithi060:osd.0> sudo journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.0.service 2023-07-20T01:23:01.527 INFO:tasks.cephadm:Deploying osd.1 on smithi136 with /dev/vg_nvme/lv_4... 2023-07-20T01:23:01.528 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-20T01:23:01.552 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:01 smithi060 ceph-mon[96524]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:23:01.552 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:01 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:01.552 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:01 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:01.552 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:01 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:01.553 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:01 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:23:01.553 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:01 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:01.553 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:01 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:01.553 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:01 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:01.553 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:01 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:23:01.559 INFO:journalctl@ceph.osd.0.smithi060.stdout:-- Logs begin at Thu 2023-07-20 01:00:02 UTC. -- 2023-07-20T01:23:01.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:01 smithi136 ceph-mon[98083]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:23:01.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:01 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:01.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:01 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:01.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:01 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:01.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:01 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:23:01.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:01 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:01.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:01 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:01.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:01 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:01.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:01 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:23:01.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:01 smithi183 ceph-mon[98477]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:23:01.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:01 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:01.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:01 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:01.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:01 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:01.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:01 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:23:01.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:01 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:01.952 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:01 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:01.952 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:01 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:01.952 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:01 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:23:02.074 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T01:23:02.357 INFO:journalctl@ceph.osd.0.smithi060.stdout:Jul 20 01:23:02 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-0[109432]: 2023-07-20T01:23:02.191+0000 7f4eaab66640 -1 osd.0 0 log_to_monitors true 2023-07-20T01:23:02.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:02 smithi136 ceph-mon[98083]: from='osd.0 [v2:172.21.15.60:6802/452810841,v1:172.21.15.60:6803/452810841]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-20T01:23:02.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:02 smithi060 ceph-mon[96524]: from='osd.0 [v2:172.21.15.60:6802/452810841,v1:172.21.15.60:6803/452810841]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-20T01:23:02.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:02 smithi183 ceph-mon[98477]: from='osd.0 [v2:172.21.15.60:6802/452810841,v1:172.21.15.60:6803/452810841]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-20T01:23:03.678 INFO:journalctl@ceph.osd.0.smithi060.stdout:Jul 20 01:23:03 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-0[109432]: 2023-07-20T01:23:03.558+0000 7f4ea01de700 -1 osd.0 0 waiting for initial osdmap 2023-07-20T01:23:03.678 INFO:journalctl@ceph.osd.0.smithi060.stdout:Jul 20 01:23:03 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-0[109432]: 2023-07-20T01:23:03.561+0000 7f4e9a7d1700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T01:23:03.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:03 smithi060 ceph-mon[96524]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:23:03.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:03 smithi060 ceph-mon[96524]: from='osd.0 [v2:172.21.15.60:6802/452810841,v1:172.21.15.60:6803/452810841]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-20T01:23:03.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:03 smithi060 ceph-mon[96524]: osdmap e6: 1 total, 0 up, 1 in 2023-07-20T01:23:03.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:03 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:23:03.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:03 smithi060 ceph-mon[96524]: from='osd.0 [v2:172.21.15.60:6802/452810841,v1:172.21.15.60:6803/452810841]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2023-07-20T01:23:03.702 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-20T01:23:03.702 INFO:teuthology.orchestra.run.smithi136.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-20T01:23:03.702 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10+0 records in 2023-07-20T01:23:03.702 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2023-07-20T01:23:03.702 INFO:teuthology.orchestra.run.smithi136.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124265 s, 844 MB/s 2023-07-20T01:23:03.702 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 2023-07-20T01:23:03.703 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping successful for: 2023-07-20T01:23:03.831 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:03 smithi136 ceph-mon[98083]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:23:03.832 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:03 smithi136 ceph-mon[98083]: from='osd.0 [v2:172.21.15.60:6802/452810841,v1:172.21.15.60:6803/452810841]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-20T01:23:03.832 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:03 smithi136 ceph-mon[98083]: osdmap e6: 1 total, 0 up, 1 in 2023-07-20T01:23:03.832 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:03 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:23:03.832 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:03 smithi136 ceph-mon[98083]: from='osd.0 [v2:172.21.15.60:6802/452810841,v1:172.21.15.60:6803/452810841]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2023-07-20T01:23:03.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:03 smithi183 ceph-mon[98477]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:23:03.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:03 smithi183 ceph-mon[98477]: from='osd.0 [v2:172.21.15.60:6802/452810841,v1:172.21.15.60:6803/452810841]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-20T01:23:03.952 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:03 smithi183 ceph-mon[98477]: osdmap e6: 1 total, 0 up, 1 in 2023-07-20T01:23:03.952 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:03 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:23:03.952 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:03 smithi183 ceph-mon[98477]: from='osd.0 [v2:172.21.15.60:6802/452810841,v1:172.21.15.60:6803/452810841]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2023-07-20T01:23:04.127 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi136:vg_nvme/lv_4 2023-07-20T01:23:04.676 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T01:23:04.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:04 smithi136 ceph-mon[98083]: from='osd.0 [v2:172.21.15.60:6802/452810841,v1:172.21.15.60:6803/452810841]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2023-07-20T01:23:04.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:04 smithi136 ceph-mon[98083]: osdmap e7: 1 total, 0 up, 1 in 2023-07-20T01:23:04.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:04 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:23:04.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:04 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:23:04.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:04 smithi136 ceph-mon[98083]: Detected new or changed devices on smithi060 2023-07-20T01:23:04.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:04 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:04.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:04 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:04.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:04 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:23:04.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:04 smithi136 ceph-mon[98083]: Adjusting osd_memory_target on smithi060 to 17173M 2023-07-20T01:23:04.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:04 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:04.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:04 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:04.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:04 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:23:04.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:04 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:04.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:04 smithi060 ceph-mon[96524]: from='osd.0 [v2:172.21.15.60:6802/452810841,v1:172.21.15.60:6803/452810841]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2023-07-20T01:23:04.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:04 smithi060 ceph-mon[96524]: osdmap e7: 1 total, 0 up, 1 in 2023-07-20T01:23:04.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:04 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:23:04.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:04 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:23:04.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:04 smithi060 ceph-mon[96524]: Detected new or changed devices on smithi060 2023-07-20T01:23:04.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:04 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:04.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:04 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:04.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:04 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:23:04.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:04 smithi060 ceph-mon[96524]: Adjusting osd_memory_target on smithi060 to 17173M 2023-07-20T01:23:04.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:04 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:04.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:04 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:04.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:04 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:23:04.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:04 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:04.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:04 smithi183 ceph-mon[98477]: from='osd.0 [v2:172.21.15.60:6802/452810841,v1:172.21.15.60:6803/452810841]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2023-07-20T01:23:04.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:04 smithi183 ceph-mon[98477]: osdmap e7: 1 total, 0 up, 1 in 2023-07-20T01:23:04.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:04 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:23:04.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:04 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:23:04.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:04 smithi183 ceph-mon[98477]: Detected new or changed devices on smithi060 2023-07-20T01:23:04.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:04 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:04.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:04 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:04.952 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:04 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:23:04.952 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:04 smithi183 ceph-mon[98477]: Adjusting osd_memory_target on smithi060 to 17173M 2023-07-20T01:23:04.952 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:04 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:04.952 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:04 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:04.952 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:04 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:23:04.952 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:04 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:05.796 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:05 smithi136 ceph-mon[98083]: purged_snaps scrub starts 2023-07-20T01:23:05.796 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:05 smithi136 ceph-mon[98083]: purged_snaps scrub ok 2023-07-20T01:23:05.796 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:05 smithi136 ceph-mon[98083]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:23:05.797 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:05 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:23:05.797 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:05 smithi136 ceph-mon[98083]: osd.0 [v2:172.21.15.60:6802/452810841,v1:172.21.15.60:6803/452810841] boot 2023-07-20T01:23:05.797 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:05 smithi136 ceph-mon[98083]: osdmap e8: 1 total, 1 up, 1 in 2023-07-20T01:23:05.798 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:05 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:23:05.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:05 smithi060 ceph-mon[96524]: purged_snaps scrub starts 2023-07-20T01:23:05.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:05 smithi060 ceph-mon[96524]: purged_snaps scrub ok 2023-07-20T01:23:05.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:05 smithi060 ceph-mon[96524]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:23:05.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:05 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:23:05.930 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:05 smithi060 ceph-mon[96524]: osd.0 [v2:172.21.15.60:6802/452810841,v1:172.21.15.60:6803/452810841] boot 2023-07-20T01:23:05.930 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:05 smithi060 ceph-mon[96524]: osdmap e8: 1 total, 1 up, 1 in 2023-07-20T01:23:05.930 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:05 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:23:05.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:05 smithi183 ceph-mon[98477]: purged_snaps scrub starts 2023-07-20T01:23:05.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:05 smithi183 ceph-mon[98477]: purged_snaps scrub ok 2023-07-20T01:23:05.952 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:05 smithi183 ceph-mon[98477]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-20T01:23:05.952 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:05 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:23:05.952 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:05 smithi183 ceph-mon[98477]: osd.0 [v2:172.21.15.60:6802/452810841,v1:172.21.15.60:6803/452810841] boot 2023-07-20T01:23:05.953 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:05 smithi183 ceph-mon[98477]: osdmap e8: 1 total, 1 up, 1 in 2023-07-20T01:23:05.953 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:05 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-20T01:23:05.997 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.015+0000 7f9b65cb3700 1 -- 172.21.15.136:0/2343925989 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b600ff700 msgr2=0x7f9b600ffb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:23:05.998 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.015+0000 7f9b65cb3700 1 --2- 172.21.15.136:0/2343925989 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b600ff700 0x7f9b600ffb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9b540099c0 tx=0x7f9b54009cd0 comp rx=0 tx=0).stop 2023-07-20T01:23:05.998 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.015+0000 7f9b65cb3700 1 -- 172.21.15.136:0/2343925989 shutdown_connections 2023-07-20T01:23:05.998 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.015+0000 7f9b65cb3700 1 --2- 172.21.15.136:0/2343925989 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b601012a0 0x7f9b60107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:23:05.998 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.015+0000 7f9b65cb3700 1 --2- 172.21.15.136:0/2343925989 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b600ff700 0x7f9b600ffb20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:23:05.998 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.015+0000 7f9b65cb3700 1 --2- 172.21.15.136:0/2343925989 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b60100900 0x7f9b60100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:23:05.998 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.015+0000 7f9b65cb3700 1 -- 172.21.15.136:0/2343925989 >> 172.21.15.136:0/2343925989 conn(0x7f9b600fac80 msgr2=0x7f9b600fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:23:05.998 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.015+0000 7f9b65cb3700 1 -- 172.21.15.136:0/2343925989 shutdown_connections 2023-07-20T01:23:05.999 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.015+0000 7f9b65cb3700 1 -- 172.21.15.136:0/2343925989 wait complete. 2023-07-20T01:23:05.999 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.016+0000 7f9b65cb3700 1 Processor -- start 2023-07-20T01:23:05.999 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.016+0000 7f9b65cb3700 1 -- start start 2023-07-20T01:23:06.000 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.017+0000 7f9b65cb3700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b600ff700 0x7f9b60068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:23:06.001 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.017+0000 7f9b65cb3700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b60100900 0x7f9b6019b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:23:06.001 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.017+0000 7f9b65cb3700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b601012a0 0x7f9b6019d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:23:06.001 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.017+0000 7f9b65cb3700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6019df90 con 0x7f9b60100900 2023-07-20T01:23:06.001 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.017+0000 7f9b65cb3700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6019e100 con 0x7f9b601012a0 2023-07-20T01:23:06.001 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.017+0000 7f9b65cb3700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6019e2c0 con 0x7f9b600ff700 2023-07-20T01:23:06.001 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.017+0000 7f9b5ffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b601012a0 0x7f9b6019d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:23:06.001 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.017+0000 7f9b5ffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b601012a0 0x7f9b6019d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37944/0 (socket says 172.21.15.136:37944) 2023-07-20T01:23:06.002 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.017+0000 7f9b5ffff700 1 -- 172.21.15.136:0/1735163287 learned_addr learned my addr 172.21.15.136:0/1735163287 (peer_addr_for_me v2:172.21.15.136:0/0) 2023-07-20T01:23:06.002 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.017+0000 7f9b5f7fe700 1 --2- 172.21.15.136:0/1735163287 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b600ff700 0x7f9b60068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:23:06.002 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.017+0000 7f9b5effd700 1 --2- 172.21.15.136:0/1735163287 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b60100900 0x7f9b6019b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:23:06.002 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.017+0000 7f9b5ffff700 1 -- 172.21.15.136:0/1735163287 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b600ff700 msgr2=0x7f9b60068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:23:06.002 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.017+0000 7f9b5ffff700 1 --2- 172.21.15.136:0/1735163287 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b600ff700 0x7f9b60068db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:23:06.002 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.017+0000 7f9b5ffff700 1 -- 172.21.15.136:0/1735163287 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b60100900 msgr2=0x7f9b6019b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:23:06.002 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.017+0000 7f9b5ffff700 1 --2- 172.21.15.136:0/1735163287 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b60100900 0x7f9b6019b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:23:06.002 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.017+0000 7f9b5ffff700 1 -- 172.21.15.136:0/1735163287 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b540095f0 con 0x7f9b601012a0 2023-07-20T01:23:06.003 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.017+0000 7f9b5ffff700 1 --2- 172.21.15.136:0/1735163287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b601012a0 0x7f9b6019d7f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9b58007ee0 tx=0x7f9b5800d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:23:06.003 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.017+0000 7f9b5cff9700 1 -- 172.21.15.136:0/1735163287 <== mon.2 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b58017070 con 0x7f9b601012a0 2023-07-20T01:23:06.003 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.017+0000 7f9b65cb3700 1 -- 172.21.15.136:0/1735163287 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b6019e5a0 con 0x7f9b601012a0 2023-07-20T01:23:06.003 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.017+0000 7f9b5cff9700 1 -- 172.21.15.136:0/1735163287 <== mon.2 v2:172.21.15.136:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9b5800f040 con 0x7f9b601012a0 2023-07-20T01:23:06.003 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.017+0000 7f9b5cff9700 1 -- 172.21.15.136:0/1735163287 <== mon.2 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b58005710 con 0x7f9b601012a0 2023-07-20T01:23:06.003 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.017+0000 7f9b65cb3700 1 -- 172.21.15.136:0/1735163287 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b6019eaa0 con 0x7f9b601012a0 2023-07-20T01:23:06.003 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.018+0000 7f9b65cb3700 1 -- 172.21.15.136:0/1735163287 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b60106dd0 con 0x7f9b601012a0 2023-07-20T01:23:06.004 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.020+0000 7f9b5cff9700 1 -- 172.21.15.136:0/1735163287 <== mon.2 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f9b58023020 con 0x7f9b601012a0 2023-07-20T01:23:06.004 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.021+0000 7f9b5cff9700 1 --2- 172.21.15.136:0/1735163287 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f9b480a1140 0x7f9b480a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:23:06.004 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.021+0000 7f9b5cff9700 1 -- 172.21.15.136:0/1735163287 <== mon.2 v2:172.21.15.136:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f9b580ca6e0 con 0x7f9b601012a0 2023-07-20T01:23:06.005 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.021+0000 7f9b5f7fe700 1 --2- 172.21.15.136:0/1735163287 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f9b480a1140 0x7f9b480a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:23:06.005 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.021+0000 7f9b5f7fe700 1 --2- 172.21.15.136:0/1735163287 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f9b480a1140 0x7f9b480a3600 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9b54000c00 tx=0x7f9b54005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:23:06.005 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.024+0000 7f9b5cff9700 1 -- 172.21.15.136:0/1735163287 <== mon.2 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9b580958a0 con 0x7f9b601012a0 2023-07-20T01:23:06.197 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:06.216+0000 7f9b65cb3700 1 -- 172.21.15.136:0/1735163287 --> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f9b60060bb0 con 0x7f9b480a1140 2023-07-20T01:23:07.028 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:06 smithi136 ceph-mon[98083]: osdmap e9: 1 total, 1 up, 1 in 2023-07-20T01:23:07.028 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:06 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:23:07.029 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:06 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:23:07.029 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:06 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:07.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:06 smithi060 ceph-mon[96524]: osdmap e9: 1 total, 1 up, 1 in 2023-07-20T01:23:07.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:06 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:23:07.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:06 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:23:07.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:06 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:07.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:06 smithi183 ceph-mon[98477]: osdmap e9: 1 total, 1 up, 1 in 2023-07-20T01:23:07.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:06 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:23:07.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:06 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:23:07.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:06 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:08.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:07 smithi136 ceph-mon[98083]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:08.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:07 smithi136 ceph-mon[98083]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:23:08.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:07 smithi060 ceph-mon[96524]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:08.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:07 smithi060 ceph-mon[96524]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:23:08.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:07 smithi183 ceph-mon[98477]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:08.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:07 smithi183 ceph-mon[98477]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:23:09.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:08 smithi136 ceph-mon[98083]: from='client.? 172.21.15.136:0/1111687736' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74dc92bd-d5ac-49ed-9918-3a91c53e9486"}]: dispatch 2023-07-20T01:23:09.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:08 smithi136 ceph-mon[98083]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74dc92bd-d5ac-49ed-9918-3a91c53e9486"}]: dispatch 2023-07-20T01:23:09.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:08 smithi136 ceph-mon[98083]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "74dc92bd-d5ac-49ed-9918-3a91c53e9486"}]': finished 2023-07-20T01:23:09.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:08 smithi136 ceph-mon[98083]: osdmap e10: 2 total, 1 up, 2 in 2023-07-20T01:23:09.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:08 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T01:23:09.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:08 smithi060 ceph-mon[96524]: from='client.? 172.21.15.136:0/1111687736' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74dc92bd-d5ac-49ed-9918-3a91c53e9486"}]: dispatch 2023-07-20T01:23:09.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:08 smithi060 ceph-mon[96524]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74dc92bd-d5ac-49ed-9918-3a91c53e9486"}]: dispatch 2023-07-20T01:23:09.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:08 smithi060 ceph-mon[96524]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "74dc92bd-d5ac-49ed-9918-3a91c53e9486"}]': finished 2023-07-20T01:23:09.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:08 smithi060 ceph-mon[96524]: osdmap e10: 2 total, 1 up, 2 in 2023-07-20T01:23:09.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:08 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T01:23:09.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:08 smithi183 ceph-mon[98477]: from='client.? 172.21.15.136:0/1111687736' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74dc92bd-d5ac-49ed-9918-3a91c53e9486"}]: dispatch 2023-07-20T01:23:09.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:08 smithi183 ceph-mon[98477]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74dc92bd-d5ac-49ed-9918-3a91c53e9486"}]: dispatch 2023-07-20T01:23:09.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:08 smithi183 ceph-mon[98477]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "74dc92bd-d5ac-49ed-9918-3a91c53e9486"}]': finished 2023-07-20T01:23:09.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:08 smithi183 ceph-mon[98477]: osdmap e10: 2 total, 1 up, 2 in 2023-07-20T01:23:09.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:08 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T01:23:10.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:09 smithi136 ceph-mon[98083]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:10.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:09 smithi136 ceph-mon[98083]: from='client.? 172.21.15.136:0/1566876659' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:23:10.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:09 smithi060 ceph-mon[96524]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:10.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:09 smithi060 ceph-mon[96524]: from='client.? 172.21.15.136:0/1566876659' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:23:10.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:09 smithi183 ceph-mon[98477]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:10.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:09 smithi183 ceph-mon[98477]: from='client.? 172.21.15.136:0/1566876659' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:23:12.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:11 smithi136 ceph-mon[98083]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:12.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:11 smithi060 ceph-mon[96524]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:12.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:11 smithi183 ceph-mon[98477]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:14.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:13 smithi136 ceph-mon[98083]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:14.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:13 smithi060 ceph-mon[96524]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:14.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:13 smithi183 ceph-mon[98477]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:16.029 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:15 smithi136 ceph-mon[98083]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:16.029 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:15 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-20T01:23:16.029 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:15 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:16.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:15 smithi060 ceph-mon[96524]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:16.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:15 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-20T01:23:16.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:15 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:16.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:15 smithi183 ceph-mon[98477]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:16.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:15 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-20T01:23:16.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:15 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:16.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:16 smithi136 ceph-mon[98083]: Deploying daemon osd.1 on smithi136 2023-07-20T01:23:17.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:16 smithi060 ceph-mon[96524]: Deploying daemon osd.1 on smithi136 2023-07-20T01:23:17.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:16 smithi183 ceph-mon[98477]: Deploying daemon osd.1 on smithi136 2023-07-20T01:23:17.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:17 smithi136 ceph-mon[98083]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:18.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:17 smithi060 ceph-mon[96524]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:18.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:17 smithi183 ceph-mon[98477]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:19.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:19 smithi136 ceph-mon[98083]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:19.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:19 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:19.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:19 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:23:19.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:19 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:20.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:19 smithi060 ceph-mon[96524]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:20.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:19 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:20.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:19 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:23:20.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:19 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:20.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:19 smithi183 ceph-mon[98477]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:20.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:19 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:20.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:19 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:23:20.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:19 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:21.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:21 smithi136 ceph-mon[98083]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:22.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:21 smithi060 ceph-mon[96524]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:22.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:21 smithi183 ceph-mon[98477]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:22.663 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:22.682+0000 7f9b5cff9700 1 -- 172.21.15.136:0/1735163287 <== mgr.14152 v2:172.21.15.60:6800/1961833180 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9b60060bb0 con 0x7f9b480a1140 2023-07-20T01:23:22.663 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 1 on host 'smithi136' 2023-07-20T01:23:22.665 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:22.684+0000 7f9b65cb3700 1 -- 172.21.15.136:0/1735163287 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f9b480a1140 msgr2=0x7f9b480a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:23:22.665 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:22.684+0000 7f9b65cb3700 1 --2- 172.21.15.136:0/1735163287 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f9b480a1140 0x7f9b480a3600 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9b54000c00 tx=0x7f9b54005fb0 comp rx=0 tx=0).stop 2023-07-20T01:23:22.665 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:22.684+0000 7f9b65cb3700 1 -- 172.21.15.136:0/1735163287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b601012a0 msgr2=0x7f9b6019d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:23:22.666 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:22.684+0000 7f9b65cb3700 1 --2- 172.21.15.136:0/1735163287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b601012a0 0x7f9b6019d7f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9b58007ee0 tx=0x7f9b5800d3b0 comp rx=0 tx=0).stop 2023-07-20T01:23:22.666 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:22.684+0000 7f9b65cb3700 1 -- 172.21.15.136:0/1735163287 shutdown_connections 2023-07-20T01:23:22.666 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:22.684+0000 7f9b65cb3700 1 --2- 172.21.15.136:0/1735163287 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f9b480a1140 0x7f9b480a3600 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:23:22.666 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:22.684+0000 7f9b65cb3700 1 --2- 172.21.15.136:0/1735163287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b601012a0 0x7f9b6019d7f0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:23:22.666 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:22.684+0000 7f9b65cb3700 1 --2- 172.21.15.136:0/1735163287 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b600ff700 0x7f9b60068db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:23:22.666 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:22.684+0000 7f9b65cb3700 1 --2- 172.21.15.136:0/1735163287 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b60100900 0x7f9b6019b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:23:22.666 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:22.684+0000 7f9b65cb3700 1 -- 172.21.15.136:0/1735163287 >> 172.21.15.136:0/1735163287 conn(0x7f9b600fac80 msgr2=0x7f9b60103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:23:22.666 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:22.684+0000 7f9b65cb3700 1 -- 172.21.15.136:0/1735163287 shutdown_connections 2023-07-20T01:23:22.667 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:23:22.684+0000 7f9b65cb3700 1 -- 172.21.15.136:0/1735163287 wait complete. 2023-07-20T01:23:23.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:22 smithi136 ceph-mon[98083]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:23.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:22 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:23.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:22 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:23.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:22 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:23.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:22 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:23:23.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:22 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:23.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:22 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:23.150 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:22 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:23.150 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:22 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:23:23.169 DEBUG:teuthology.orchestra.run.smithi136:osd.1> sudo journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.1.service 2023-07-20T01:23:23.172 INFO:tasks.cephadm:Deploying osd.2 on smithi183 with /dev/vg_nvme/lv_4... 2023-07-20T01:23:23.173 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-20T01:23:23.205 INFO:journalctl@ceph.osd.1.smithi136.stdout:-- Logs begin at Thu 2023-07-20 01:00:24 UTC. -- 2023-07-20T01:23:23.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:22 smithi060 ceph-mon[96524]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:23.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:22 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:23.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:22 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:23.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:22 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:23.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:22 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:23:23.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:22 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:23.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:22 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:23.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:22 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:23.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:22 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:23:23.450 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:22 smithi183 ceph-mon[98477]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:23.451 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:22 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:23.451 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:22 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:23.451 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:22 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:23.452 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:22 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:23:23.452 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:22 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:23.452 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:22 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:23.452 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:22 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:23.453 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:22 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:23:23.736 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T01:23:23.896 INFO:journalctl@ceph.osd.1.smithi136.stdout:Jul 20 01:23:23 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-1[103151]: 2023-07-20T01:23:23.551+0000 7f2152f32640 -1 osd.1 0 log_to_monitors true 2023-07-20T01:23:24.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:23 smithi136 ceph-mon[98083]: from='osd.1 [v2:172.21.15.136:6800/2541801111,v1:172.21.15.136:6801/2541801111]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T01:23:24.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:23 smithi136 ceph-mon[98083]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T01:23:24.291 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:23 smithi183 ceph-mon[98477]: from='osd.1 [v2:172.21.15.136:6800/2541801111,v1:172.21.15.136:6801/2541801111]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T01:23:24.291 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:23 smithi183 ceph-mon[98477]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T01:23:24.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:23 smithi060 ceph-mon[96524]: from='osd.1 [v2:172.21.15.136:6800/2541801111,v1:172.21.15.136:6801/2541801111]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T01:23:24.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:23 smithi060 ceph-mon[96524]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-20T01:23:25.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:25 smithi183 ceph-mon[98477]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:25.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:25 smithi183 ceph-mon[98477]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-20T01:23:25.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:25 smithi183 ceph-mon[98477]: osdmap e11: 2 total, 1 up, 2 in 2023-07-20T01:23:25.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:25 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T01:23:25.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:25 smithi183 ceph-mon[98477]: from='osd.1 [v2:172.21.15.136:6800/2541801111,v1:172.21.15.136:6801/2541801111]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-07-20T01:23:25.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:25 smithi183 ceph-mon[98477]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-07-20T01:23:25.235 INFO:journalctl@ceph.osd.1.smithi136.stdout:Jul 20 01:23:25 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-1[103151]: 2023-07-20T01:23:25.005+0000 7f21485aa700 -1 osd.1 0 waiting for initial osdmap 2023-07-20T01:23:25.235 INFO:journalctl@ceph.osd.1.smithi136.stdout:Jul 20 01:23:25 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-1[103151]: 2023-07-20T01:23:25.008+0000 7f21443a0700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T01:23:25.236 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:25 smithi136 ceph-mon[98083]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:25.236 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:25 smithi136 ceph-mon[98083]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-20T01:23:25.236 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:25 smithi136 ceph-mon[98083]: osdmap e11: 2 total, 1 up, 2 in 2023-07-20T01:23:25.236 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T01:23:25.236 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:25 smithi136 ceph-mon[98083]: from='osd.1 [v2:172.21.15.136:6800/2541801111,v1:172.21.15.136:6801/2541801111]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-07-20T01:23:25.236 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:25 smithi136 ceph-mon[98083]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-07-20T01:23:25.337 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-20T01:23:25.338 INFO:teuthology.orchestra.run.smithi183.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-20T01:23:25.338 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10+0 records in 2023-07-20T01:23:25.338 INFO:teuthology.orchestra.run.smithi183.stderr:10+0 records out 2023-07-20T01:23:25.338 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0137531 s, 762 MB/s 2023-07-20T01:23:25.338 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping successful for: 2023-07-20T01:23:25.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:25 smithi060 ceph-mon[96524]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:25.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:25 smithi060 ceph-mon[96524]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-20T01:23:25.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:25 smithi060 ceph-mon[96524]: osdmap e11: 2 total, 1 up, 2 in 2023-07-20T01:23:25.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:25 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T01:23:25.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:25 smithi060 ceph-mon[96524]: from='osd.1 [v2:172.21.15.136:6800/2541801111,v1:172.21.15.136:6801/2541801111]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-07-20T01:23:25.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:25 smithi060 ceph-mon[96524]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-07-20T01:23:25.674 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi183:vg_nvme/lv_4 2023-07-20T01:23:26.227 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T01:23:26.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:26 smithi136 ceph-mon[98083]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2023-07-20T01:23:26.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:26 smithi136 ceph-mon[98083]: osdmap e12: 2 total, 1 up, 2 in 2023-07-20T01:23:26.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:26 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T01:23:26.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:26 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T01:23:26.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:26 smithi136 ceph-mon[98083]: Detected new or changed devices on smithi136 2023-07-20T01:23:26.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:26 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:26.399 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:26 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:26.399 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:26 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:23:26.399 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:26 smithi136 ceph-mon[98083]: Adjusting osd_memory_target on smithi136 to 17168M 2023-07-20T01:23:26.399 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:26 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:26.399 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:26 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:26.399 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:26 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:23:26.399 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:26 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:26.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:26 smithi060 ceph-mon[96524]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2023-07-20T01:23:26.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:26 smithi060 ceph-mon[96524]: osdmap e12: 2 total, 1 up, 2 in 2023-07-20T01:23:26.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:26 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T01:23:26.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:26 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T01:23:26.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:26 smithi060 ceph-mon[96524]: Detected new or changed devices on smithi136 2023-07-20T01:23:26.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:26 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:26.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:26 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:26.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:26 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:23:26.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:26 smithi060 ceph-mon[96524]: Adjusting osd_memory_target on smithi136 to 17168M 2023-07-20T01:23:26.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:26 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:26.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:26 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:26.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:26 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:23:26.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:26 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:26.446 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:26 smithi183 ceph-mon[98477]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2023-07-20T01:23:26.446 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:26 smithi183 ceph-mon[98477]: osdmap e12: 2 total, 1 up, 2 in 2023-07-20T01:23:26.447 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:26 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T01:23:26.447 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:26 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T01:23:26.447 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:26 smithi183 ceph-mon[98477]: Detected new or changed devices on smithi136 2023-07-20T01:23:26.447 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:26 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:26.447 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:26 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:26.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:26 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:23:26.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:26 smithi183 ceph-mon[98477]: Adjusting osd_memory_target on smithi136 to 17168M 2023-07-20T01:23:26.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:26 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:26.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:26 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:26.491 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:26 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:23:26.491 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:26 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:27.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:27 smithi183 ceph-mon[98477]: purged_snaps scrub starts 2023-07-20T01:23:27.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:27 smithi183 ceph-mon[98477]: purged_snaps scrub ok 2023-07-20T01:23:27.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:27 smithi183 ceph-mon[98477]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:27.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:27 smithi183 ceph-mon[98477]: osd.1 [v2:172.21.15.136:6800/2541801111,v1:172.21.15.136:6801/2541801111] boot 2023-07-20T01:23:27.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:27 smithi183 ceph-mon[98477]: osdmap e13: 2 total, 2 up, 2 in 2023-07-20T01:23:27.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:27 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T01:23:27.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:27 smithi136 ceph-mon[98083]: purged_snaps scrub starts 2023-07-20T01:23:27.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:27 smithi136 ceph-mon[98083]: purged_snaps scrub ok 2023-07-20T01:23:27.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:27 smithi136 ceph-mon[98083]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:27.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:27 smithi136 ceph-mon[98083]: osd.1 [v2:172.21.15.136:6800/2541801111,v1:172.21.15.136:6801/2541801111] boot 2023-07-20T01:23:27.399 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:27 smithi136 ceph-mon[98083]: osdmap e13: 2 total, 2 up, 2 in 2023-07-20T01:23:27.399 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:27 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T01:23:27.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:27 smithi060 ceph-mon[96524]: purged_snaps scrub starts 2023-07-20T01:23:27.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:27 smithi060 ceph-mon[96524]: purged_snaps scrub ok 2023-07-20T01:23:27.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:27 smithi060 ceph-mon[96524]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-20T01:23:27.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:27 smithi060 ceph-mon[96524]: osd.1 [v2:172.21.15.136:6800/2541801111,v1:172.21.15.136:6801/2541801111] boot 2023-07-20T01:23:27.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:27 smithi060 ceph-mon[96524]: osdmap e13: 2 total, 2 up, 2 in 2023-07-20T01:23:27.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:27 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-20T01:23:27.444 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.463+0000 7fc0d9745700 1 -- 172.21.15.183:0/1262204840 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0d40ff710 msgr2=0x7fc0d40ffb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:23:27.445 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.463+0000 7fc0d9745700 1 --2- 172.21.15.183:0/1262204840 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0d40ff710 0x7fc0d40ffb30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc009990 tx=0x7fc0bc009ca0 comp rx=0 tx=0).stop 2023-07-20T01:23:27.445 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.463+0000 7fc0d9745700 1 -- 172.21.15.183:0/1262204840 shutdown_connections 2023-07-20T01:23:27.445 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.463+0000 7fc0d9745700 1 --2- 172.21.15.183:0/1262204840 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0d41012b0 0x7fc0d4107b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:23:27.445 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.463+0000 7fc0d9745700 1 --2- 172.21.15.183:0/1262204840 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0d40ff710 0x7fc0d40ffb30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:23:27.445 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.463+0000 7fc0d9745700 1 --2- 172.21.15.183:0/1262204840 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc0d4100910 0x7fc0d4100d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:23:27.445 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.463+0000 7fc0d9745700 1 -- 172.21.15.183:0/1262204840 >> 172.21.15.183:0/1262204840 conn(0x7fc0d40facb0 msgr2=0x7fc0d40fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:23:27.446 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.463+0000 7fc0d9745700 1 -- 172.21.15.183:0/1262204840 shutdown_connections 2023-07-20T01:23:27.446 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.463+0000 7fc0d9745700 1 -- 172.21.15.183:0/1262204840 wait complete. 2023-07-20T01:23:27.448 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.464+0000 7fc0d9745700 1 Processor -- start 2023-07-20T01:23:27.448 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.464+0000 7fc0d9745700 1 -- start start 2023-07-20T01:23:27.448 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.464+0000 7fc0d9745700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc0d40ff710 0x7fc0d4069730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:23:27.449 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.464+0000 7fc0d9745700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0d4100910 0x7fc0d419b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:23:27.449 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.464+0000 7fc0d9745700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0d41012b0 0x7fc0d419d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:23:27.449 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.464+0000 7fc0d9745700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d419df60 con 0x7fc0d41012b0 2023-07-20T01:23:27.449 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.464+0000 7fc0d9745700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d419e0d0 con 0x7fc0d40ff710 2023-07-20T01:23:27.450 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.464+0000 7fc0d9745700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d419e290 con 0x7fc0d4100910 2023-07-20T01:23:27.450 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.464+0000 7fc0d27fc700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0d4100910 0x7fc0d419b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:23:27.450 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.464+0000 7fc0d27fc700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0d4100910 0x7fc0d419b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:45862/0 (socket says 172.21.15.183:45862) 2023-07-20T01:23:27.450 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.464+0000 7fc0d27fc700 1 -- 172.21.15.183:0/485021764 learned_addr learned my addr 172.21.15.183:0/485021764 (peer_addr_for_me v2:172.21.15.183:0/0) 2023-07-20T01:23:27.451 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.464+0000 7fc0d2ffd700 1 --2- 172.21.15.183:0/485021764 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc0d40ff710 0x7fc0d4069730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:23:27.451 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.464+0000 7fc0d37fe700 1 --2- 172.21.15.183:0/485021764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0d41012b0 0x7fc0d419d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:23:27.451 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.465+0000 7fc0d27fc700 1 -- 172.21.15.183:0/485021764 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc0d40ff710 msgr2=0x7fc0d4069730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:23:27.451 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.465+0000 7fc0d27fc700 1 --2- 172.21.15.183:0/485021764 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc0d40ff710 0x7fc0d4069730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:23:27.452 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.465+0000 7fc0d27fc700 1 -- 172.21.15.183:0/485021764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0d41012b0 msgr2=0x7fc0d419d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:23:27.452 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.465+0000 7fc0d27fc700 1 --2- 172.21.15.183:0/485021764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0d41012b0 0x7fc0d419d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:23:27.452 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.465+0000 7fc0d27fc700 1 -- 172.21.15.183:0/485021764 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0bc0095f0 con 0x7fc0d4100910 2023-07-20T01:23:27.452 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.465+0000 7fc0d27fc700 1 --2- 172.21.15.183:0/485021764 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0d4100910 0x7fc0d419b170 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc0c400c9b0 tx=0x7fc0c400ccc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:23:27.453 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.465+0000 7fc0cbfff700 1 -- 172.21.15.183:0/485021764 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c4007b00 con 0x7fc0d4100910 2023-07-20T01:23:27.453 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.465+0000 7fc0cbfff700 1 -- 172.21.15.183:0/485021764 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc0c400ce50 con 0x7fc0d4100910 2023-07-20T01:23:27.453 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.465+0000 7fc0cbfff700 1 -- 172.21.15.183:0/485021764 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c4018420 con 0x7fc0d4100910 2023-07-20T01:23:27.453 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.465+0000 7fc0d9745700 1 -- 172.21.15.183:0/485021764 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0d419e570 con 0x7fc0d4100910 2023-07-20T01:23:27.454 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.465+0000 7fc0d9745700 1 -- 172.21.15.183:0/485021764 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0d419eaa0 con 0x7fc0d4100910 2023-07-20T01:23:27.454 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.466+0000 7fc0d9745700 1 -- 172.21.15.183:0/485021764 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0d4106d00 con 0x7fc0d4100910 2023-07-20T01:23:27.454 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.467+0000 7fc0cbfff700 1 -- 172.21.15.183:0/485021764 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7fc0c400f450 con 0x7fc0d4100910 2023-07-20T01:23:27.455 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.467+0000 7fc0cbfff700 1 --2- 172.21.15.183:0/485021764 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fc0c00a10c0 0x7fc0c00a3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:23:27.455 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.467+0000 7fc0cbfff700 1 -- 172.21.15.183:0/485021764 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7fc0c40cafc0 con 0x7fc0d4100910 2023-07-20T01:23:27.456 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.468+0000 7fc0d2ffd700 1 --2- 172.21.15.183:0/485021764 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fc0c00a10c0 0x7fc0c00a3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:23:27.456 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.468+0000 7fc0d2ffd700 1 --2- 172.21.15.183:0/485021764 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fc0c00a10c0 0x7fc0c00a3580 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc00b5c0 tx=0x7fc0bc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:23:27.457 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.472+0000 7fc0cbfff700 1 -- 172.21.15.183:0/485021764 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc0c4095f60 con 0x7fc0d4100910 2023-07-20T01:23:27.642 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:27.661+0000 7fc0d9745700 1 -- 172.21.15.183:0/485021764 --> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fc0d4060bb0 con 0x7fc0c00a10c0 2023-07-20T01:23:28.632 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:28 smithi183 ceph-mon[98477]: osdmap e14: 2 total, 2 up, 2 in 2023-07-20T01:23:28.632 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:28 smithi183 ceph-mon[98477]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:23:28.632 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:28 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:23:28.632 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:28 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:23:28.632 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:28 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:28.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:28 smithi136 ceph-mon[98083]: osdmap e14: 2 total, 2 up, 2 in 2023-07-20T01:23:28.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:28 smithi136 ceph-mon[98083]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:23:28.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:28 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:23:28.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:28 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:23:28.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:28 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:28.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:28 smithi060 ceph-mon[96524]: osdmap e14: 2 total, 2 up, 2 in 2023-07-20T01:23:28.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:28 smithi060 ceph-mon[96524]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:23:28.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:28 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:23:28.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:28 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:23:28.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:28 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:29.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:29 smithi136 ceph-mon[98083]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:29.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:29 smithi060 ceph-mon[96524]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:29.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:29 smithi183 ceph-mon[98477]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:30.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:30 smithi136 ceph-mon[98083]: from='client.? 172.21.15.183:0/1716042583' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c33a6742-8287-4fb4-b707-7e0d9660fd53"}]: dispatch 2023-07-20T01:23:30.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:30 smithi136 ceph-mon[98083]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c33a6742-8287-4fb4-b707-7e0d9660fd53"}]: dispatch 2023-07-20T01:23:30.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:30 smithi136 ceph-mon[98083]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c33a6742-8287-4fb4-b707-7e0d9660fd53"}]': finished 2023-07-20T01:23:30.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:30 smithi136 ceph-mon[98083]: osdmap e15: 3 total, 2 up, 3 in 2023-07-20T01:23:30.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:30 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:23:30.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:30 smithi060 ceph-mon[96524]: from='client.? 172.21.15.183:0/1716042583' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c33a6742-8287-4fb4-b707-7e0d9660fd53"}]: dispatch 2023-07-20T01:23:30.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:30 smithi060 ceph-mon[96524]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c33a6742-8287-4fb4-b707-7e0d9660fd53"}]: dispatch 2023-07-20T01:23:30.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:30 smithi060 ceph-mon[96524]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c33a6742-8287-4fb4-b707-7e0d9660fd53"}]': finished 2023-07-20T01:23:30.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:30 smithi060 ceph-mon[96524]: osdmap e15: 3 total, 2 up, 3 in 2023-07-20T01:23:30.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:30 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:23:30.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:30 smithi183 ceph-mon[98477]: from='client.? 172.21.15.183:0/1716042583' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c33a6742-8287-4fb4-b707-7e0d9660fd53"}]: dispatch 2023-07-20T01:23:30.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:30 smithi183 ceph-mon[98477]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c33a6742-8287-4fb4-b707-7e0d9660fd53"}]: dispatch 2023-07-20T01:23:30.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:30 smithi183 ceph-mon[98477]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c33a6742-8287-4fb4-b707-7e0d9660fd53"}]': finished 2023-07-20T01:23:30.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:30 smithi183 ceph-mon[98477]: osdmap e15: 3 total, 2 up, 3 in 2023-07-20T01:23:30.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:30 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:23:31.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:31 smithi136 ceph-mon[98083]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:31.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:31 smithi136 ceph-mon[98083]: from='client.? 172.21.15.183:0/2176488578' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:23:31.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:31 smithi060 ceph-mon[96524]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:31.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:31 smithi060 ceph-mon[96524]: from='client.? 172.21.15.183:0/2176488578' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:23:31.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:31 smithi183 ceph-mon[98477]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:31.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:31 smithi183 ceph-mon[98477]: from='client.? 172.21.15.183:0/2176488578' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:23:33.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:33 smithi183 ceph-mon[98477]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:33.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:33 smithi136 ceph-mon[98083]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:33.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:33 smithi060 ceph-mon[96524]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:35.602 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:35 smithi183 ceph-mon[98477]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:35.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:35 smithi136 ceph-mon[98083]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:35.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:35 smithi060 ceph-mon[96524]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:37.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:37 smithi136 ceph-mon[98083]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:37.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:37 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-20T01:23:37.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:37 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:37.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:37 smithi136 ceph-mon[98083]: Deploying daemon osd.2 on smithi183 2023-07-20T01:23:37.671 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:37 smithi183 ceph-mon[98477]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:37.671 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:37 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-20T01:23:37.671 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:37 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:37.671 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:37 smithi183 ceph-mon[98477]: Deploying daemon osd.2 on smithi183 2023-07-20T01:23:37.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:37 smithi060 ceph-mon[96524]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:37.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:37 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-20T01:23:37.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:37 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:37.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:37 smithi060 ceph-mon[96524]: Deploying daemon osd.2 on smithi183 2023-07-20T01:23:39.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:39 smithi136 ceph-mon[98083]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:39.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:39 smithi060 ceph-mon[96524]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:39.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:39 smithi183 ceph-mon[98477]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:41.813 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:41 smithi183 ceph-mon[98477]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:41.813 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:41 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:41.813 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:41 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:41.813 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:41 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:23:42.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:41 smithi136 ceph-mon[98083]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:42.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:41 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:42.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:41 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:42.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:41 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:23:42.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:41 smithi060 ceph-mon[96524]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:42.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:41 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:42.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:41 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:42.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:41 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:23:43.455 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 2 on host 'smithi183' 2023-07-20T01:23:43.455 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:43.473+0000 7fc0cbfff700 1 -- 172.21.15.183:0/485021764 <== mgr.14152 v2:172.21.15.60:6800/1961833180 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc0d4060bb0 con 0x7fc0c00a10c0 2023-07-20T01:23:43.456 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:43.476+0000 7fc0d9745700 1 -- 172.21.15.183:0/485021764 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fc0c00a10c0 msgr2=0x7fc0c00a3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:23:43.456 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:43.476+0000 7fc0d9745700 1 --2- 172.21.15.183:0/485021764 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fc0c00a10c0 0x7fc0c00a3580 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc00b5c0 tx=0x7fc0bc005fd0 comp rx=0 tx=0).stop 2023-07-20T01:23:43.457 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:43.476+0000 7fc0d9745700 1 -- 172.21.15.183:0/485021764 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0d4100910 msgr2=0x7fc0d419b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:23:43.457 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:43.476+0000 7fc0d9745700 1 --2- 172.21.15.183:0/485021764 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0d4100910 0x7fc0d419b170 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc0c400c9b0 tx=0x7fc0c400ccc0 comp rx=0 tx=0).stop 2023-07-20T01:23:43.457 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:43.476+0000 7fc0d9745700 1 -- 172.21.15.183:0/485021764 shutdown_connections 2023-07-20T01:23:43.457 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:43.476+0000 7fc0d9745700 1 --2- 172.21.15.183:0/485021764 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fc0c00a10c0 0x7fc0c00a3580 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:23:43.457 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:43.476+0000 7fc0d9745700 1 --2- 172.21.15.183:0/485021764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0d41012b0 0x7fc0d419d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:23:43.457 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:43.476+0000 7fc0d9745700 1 --2- 172.21.15.183:0/485021764 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc0d40ff710 0x7fc0d4069730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:23:43.457 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:43.476+0000 7fc0d9745700 1 --2- 172.21.15.183:0/485021764 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0d4100910 0x7fc0d419b170 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:23:43.458 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:43.476+0000 7fc0d9745700 1 -- 172.21.15.183:0/485021764 >> 172.21.15.183:0/485021764 conn(0x7fc0d40facb0 msgr2=0x7fc0d4103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:23:43.458 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:43.476+0000 7fc0d9745700 1 -- 172.21.15.183:0/485021764 shutdown_connections 2023-07-20T01:23:43.458 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:23:43.476+0000 7fc0d9745700 1 -- 172.21.15.183:0/485021764 wait complete. 2023-07-20T01:23:43.918 DEBUG:teuthology.orchestra.run.smithi183:osd.2> sudo journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.2.service 2023-07-20T01:23:43.920 INFO:tasks.cephadm:Deploying osd.3 on smithi060 with /dev/vg_nvme/lv_3... 2023-07-20T01:23:43.921 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-20T01:23:43.953 INFO:journalctl@ceph.osd.2.smithi183.stdout:-- Logs begin at Thu 2023-07-20 01:00:08 UTC. -- 2023-07-20T01:23:44.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:43 smithi136 ceph-mon[98083]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:44.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:43 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:44.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:43 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:44.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:43 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:44.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:43 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:23:44.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:43 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:44.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:43 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:44.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:43 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:44.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:43 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:23:44.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:43 smithi060 ceph-mon[96524]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:44.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:43 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:44.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:43 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:44.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:43 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:44.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:43 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:23:44.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:43 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:44.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:43 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:44.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:43 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:44.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:43 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:23:44.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:43 smithi183 ceph-mon[98477]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:44.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:43 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:44.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:43 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:44.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:43 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:44.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:43 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:23:44.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:43 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:44.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:43 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:44.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:43 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:44.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:43 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:23:44.482 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:23:45.053 INFO:journalctl@ceph.osd.2.smithi183.stdout:Jul 20 01:23:44 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-2[104265]: 2023-07-20T01:23:44.758+0000 7fcc4271e640 -1 osd.2 0 log_to_monitors true 2023-07-20T01:23:45.887 INFO:teuthology.orchestra.run.smithi060.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-20T01:23:45.887 INFO:teuthology.orchestra.run.smithi060.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-20T01:23:45.887 INFO:teuthology.orchestra.run.smithi060.stderr: stderr: 10+0 records in 2023-07-20T01:23:45.888 INFO:teuthology.orchestra.run.smithi060.stderr:10+0 records out 2023-07-20T01:23:45.888 INFO:teuthology.orchestra.run.smithi060.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00804415 s, 1.3 GB/s 2023-07-20T01:23:45.888 INFO:teuthology.orchestra.run.smithi060.stderr:--> Zapping successful for: 2023-07-20T01:23:46.025 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:45 smithi060 ceph-mon[96524]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:46.026 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:45 smithi060 ceph-mon[96524]: from='osd.2 [v2:172.21.15.183:6800/4233496322,v1:172.21.15.183:6801/4233496322]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T01:23:46.026 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:45 smithi060 ceph-mon[96524]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T01:23:46.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:45 smithi136 ceph-mon[98083]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:46.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:45 smithi136 ceph-mon[98083]: from='osd.2 [v2:172.21.15.183:6800/4233496322,v1:172.21.15.183:6801/4233496322]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T01:23:46.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:45 smithi136 ceph-mon[98083]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T01:23:46.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:45 smithi183 ceph-mon[98477]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:46.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:45 smithi183 ceph-mon[98477]: from='osd.2 [v2:172.21.15.183:6800/4233496322,v1:172.21.15.183:6801/4233496322]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T01:23:46.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:45 smithi183 ceph-mon[98477]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-20T01:23:46.239 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi060:vg_nvme/lv_3 2023-07-20T01:23:46.813 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:23:46.999 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:46 smithi060 ceph-mon[96524]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-20T01:23:47.000 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:46 smithi060 ceph-mon[96524]: osdmap e16: 3 total, 2 up, 3 in 2023-07-20T01:23:47.000 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:46 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:23:47.000 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:46 smithi060 ceph-mon[96524]: from='osd.2 [v2:172.21.15.183:6800/4233496322,v1:172.21.15.183:6801/4233496322]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2023-07-20T01:23:47.000 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:46 smithi060 ceph-mon[96524]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2023-07-20T01:23:47.000 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:46 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:47.000 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:46 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:47.001 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:46 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:23:47.001 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:46 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:47.001 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:46 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:47.001 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:46 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:23:47.001 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:46 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:47.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:46 smithi136 ceph-mon[98083]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-20T01:23:47.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:46 smithi136 ceph-mon[98083]: osdmap e16: 3 total, 2 up, 3 in 2023-07-20T01:23:47.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:46 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:23:47.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:46 smithi136 ceph-mon[98083]: from='osd.2 [v2:172.21.15.183:6800/4233496322,v1:172.21.15.183:6801/4233496322]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2023-07-20T01:23:47.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:46 smithi136 ceph-mon[98083]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2023-07-20T01:23:47.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:46 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:47.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:46 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:47.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:46 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:23:47.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:46 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:47.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:46 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:47.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:46 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:23:47.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:46 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:47.200 INFO:journalctl@ceph.osd.2.smithi183.stdout:Jul 20 01:23:46 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-2[104265]: 2023-07-20T01:23:46.772+0000 7fcc37d96700 -1 osd.2 0 waiting for initial osdmap 2023-07-20T01:23:47.201 INFO:journalctl@ceph.osd.2.smithi183.stdout:Jul 20 01:23:46 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-2[104265]: 2023-07-20T01:23:46.775+0000 7fcc3338b700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T01:23:47.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:46 smithi183 ceph-mon[98477]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-20T01:23:47.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:46 smithi183 ceph-mon[98477]: osdmap e16: 3 total, 2 up, 3 in 2023-07-20T01:23:47.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:46 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:23:47.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:46 smithi183 ceph-mon[98477]: from='osd.2 [v2:172.21.15.183:6800/4233496322,v1:172.21.15.183:6801/4233496322]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2023-07-20T01:23:47.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:46 smithi183 ceph-mon[98477]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2023-07-20T01:23:47.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:46 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:47.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:46 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:47.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:46 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:23:47.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:46 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:47.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:46 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:47.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:46 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:23:47.203 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:46 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:48.046 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.064+0000 7f3972249700 1 -- 172.21.15.60:0/3219531598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f396c103c60 msgr2=0x7f396c106050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:23:48.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.064+0000 7f3972249700 1 --2- 172.21.15.60:0/3219531598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f396c103c60 0x7f396c106050 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3960009990 tx=0x7f3960009ca0 comp rx=0 tx=0).stop 2023-07-20T01:23:48.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.064+0000 7f3972249700 1 -- 172.21.15.60:0/3219531598 shutdown_connections 2023-07-20T01:23:48.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.064+0000 7f3972249700 1 --2- 172.21.15.60:0/3219531598 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f396c106590 0x7f396c108980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:23:48.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.064+0000 7f3972249700 1 --2- 172.21.15.60:0/3219531598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f396c101330 0x7f396c103720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:23:48.047 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.064+0000 7f3972249700 1 --2- 172.21.15.60:0/3219531598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f396c103c60 0x7f396c106050 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:23:48.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.064+0000 7f3972249700 1 -- 172.21.15.60:0/3219531598 >> 172.21.15.60:0/3219531598 conn(0x7f396c0fac80 msgr2=0x7f396c0fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:23:48.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.064+0000 7f3972249700 1 -- 172.21.15.60:0/3219531598 shutdown_connections 2023-07-20T01:23:48.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.064+0000 7f3972249700 1 -- 172.21.15.60:0/3219531598 wait complete. 2023-07-20T01:23:48.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.065+0000 7f3972249700 1 Processor -- start 2023-07-20T01:23:48.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.065+0000 7f3972249700 1 -- start start 2023-07-20T01:23:48.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.066+0000 7f3972249700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f396c101330 0x7f396c194200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:23:48.048 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.066+0000 7f3972249700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f396c103c60 0x7f396c194740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:23:48.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.066+0000 7f3972249700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f396c106590 0x7f396c19b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:23:48.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.066+0000 7f3972249700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396c19bd90 con 0x7f396c106590 2023-07-20T01:23:48.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.066+0000 7f3972249700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396c19bed0 con 0x7f396c103c60 2023-07-20T01:23:48.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.066+0000 7f3972249700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396c19c060 con 0x7f396c101330 2023-07-20T01:23:48.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.066+0000 7f396bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f396c106590 0x7f396c19b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:23:48.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.066+0000 7f396bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f396c106590 0x7f396c19b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42296/0 (socket says 172.21.15.60:42296) 2023-07-20T01:23:48.049 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.066+0000 7f396bfff700 1 -- 172.21.15.60:0/289143518 learned_addr learned my addr 172.21.15.60:0/289143518 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:23:48.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.066+0000 7f396b7fe700 1 --2- 172.21.15.60:0/289143518 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f396c101330 0x7f396c194200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:23:48.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.066+0000 7f396affd700 1 --2- 172.21.15.60:0/289143518 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f396c103c60 0x7f396c194740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:23:48.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.066+0000 7f396bfff700 1 -- 172.21.15.60:0/289143518 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f396c101330 msgr2=0x7f396c194200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:23:48.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.066+0000 7f396bfff700 1 --2- 172.21.15.60:0/289143518 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f396c101330 0x7f396c194200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:23:48.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.066+0000 7f396bfff700 1 -- 172.21.15.60:0/289143518 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f396c103c60 msgr2=0x7f396c194740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:23:48.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.066+0000 7f396bfff700 1 --2- 172.21.15.60:0/289143518 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f396c103c60 0x7f396c194740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:23:48.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.066+0000 7f396bfff700 1 -- 172.21.15.60:0/289143518 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39600095f0 con 0x7f396c106590 2023-07-20T01:23:48.050 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.066+0000 7f396affd700 1 --2- 172.21.15.60:0/289143518 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f396c103c60 0x7f396c194740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T01:23:48.051 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.066+0000 7f396bfff700 1 --2- 172.21.15.60:0/289143518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f396c106590 0x7f396c19b7c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f396400efc0 tx=0x7f396400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:23:48.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:47 smithi136 ceph-mon[98083]: purged_snaps scrub starts 2023-07-20T01:23:48.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:47 smithi136 ceph-mon[98083]: purged_snaps scrub ok 2023-07-20T01:23:48.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:47 smithi136 ceph-mon[98083]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:48.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:47 smithi136 ceph-mon[98083]: Detected new or changed devices on smithi183 2023-07-20T01:23:48.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:47 smithi136 ceph-mon[98083]: Adjusting osd_memory_target on smithi183 to 17168M 2023-07-20T01:23:48.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:47 smithi136 ceph-mon[98083]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2023-07-20T01:23:48.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:47 smithi136 ceph-mon[98083]: osdmap e17: 3 total, 2 up, 3 in 2023-07-20T01:23:48.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:47 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:23:48.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:47 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:23:48.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:47 smithi060 ceph-mon[96524]: purged_snaps scrub starts 2023-07-20T01:23:48.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:47 smithi060 ceph-mon[96524]: purged_snaps scrub ok 2023-07-20T01:23:48.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:47 smithi060 ceph-mon[96524]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:48.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:47 smithi060 ceph-mon[96524]: Detected new or changed devices on smithi183 2023-07-20T01:23:48.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:47 smithi060 ceph-mon[96524]: Adjusting osd_memory_target on smithi183 to 17168M 2023-07-20T01:23:48.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:47 smithi060 ceph-mon[96524]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2023-07-20T01:23:48.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:47 smithi060 ceph-mon[96524]: osdmap e17: 3 total, 2 up, 3 in 2023-07-20T01:23:48.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:47 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:23:48.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:47 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:23:48.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:47 smithi183 ceph-mon[98477]: purged_snaps scrub starts 2023-07-20T01:23:48.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:47 smithi183 ceph-mon[98477]: purged_snaps scrub ok 2023-07-20T01:23:48.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:47 smithi183 ceph-mon[98477]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:48.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:47 smithi183 ceph-mon[98477]: Detected new or changed devices on smithi183 2023-07-20T01:23:48.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:47 smithi183 ceph-mon[98477]: Adjusting osd_memory_target on smithi183 to 17168M 2023-07-20T01:23:48.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:47 smithi183 ceph-mon[98477]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2023-07-20T01:23:48.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:47 smithi183 ceph-mon[98477]: osdmap e17: 3 total, 2 up, 3 in 2023-07-20T01:23:48.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:47 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:23:48.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:47 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:23:48.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.327+0000 7f3968ff9700 1 -- 172.21.15.60:0/289143518 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3964018070 con 0x7f396c106590 2023-07-20T01:23:48.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.327+0000 7f3968ff9700 1 -- 172.21.15.60:0/289143518 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f396400f040 con 0x7f396c106590 2023-07-20T01:23:48.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.327+0000 7f3972249700 1 -- 172.21.15.60:0/289143518 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f396c19c2e0 con 0x7f396c106590 2023-07-20T01:23:48.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.327+0000 7f3968ff9700 1 -- 172.21.15.60:0/289143518 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3964003d70 con 0x7f396c106590 2023-07-20T01:23:48.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.327+0000 7f3972249700 1 -- 172.21.15.60:0/289143518 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f396c19c830 con 0x7f396c106590 2023-07-20T01:23:48.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.328+0000 7f3972249700 1 -- 172.21.15.60:0/289143518 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f396c0fc800 con 0x7f396c106590 2023-07-20T01:23:48.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.330+0000 7f3968ff9700 1 -- 172.21.15.60:0/289143518 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f396400ca50 con 0x7f396c106590 2023-07-20T01:23:48.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.330+0000 7f3968ff9700 1 --2- 172.21.15.60:0/289143518 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f39540a1140 0x7f39540a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:23:48.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.331+0000 7f3968ff9700 1 -- 172.21.15.60:0/289143518 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 2851+0+0 (secure 0 0 0) 0x7f39640cb840 con 0x7f396c106590 2023-07-20T01:23:48.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.331+0000 7f396b7fe700 1 --2- 172.21.15.60:0/289143518 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f39540a1140 0x7f39540a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:23:48.312 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.331+0000 7f396b7fe700 1 --2- 172.21.15.60:0/289143518 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f39540a1140 0x7f39540a3600 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f395c009c60 tx=0x7f395c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:23:48.314 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.334+0000 7f3968ff9700 1 -- 172.21.15.60:0/289143518 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3964096650 con 0x7f396c106590 2023-07-20T01:23:48.520 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:48.539+0000 7f3972249700 1 -- 172.21.15.60:0/289143518 --> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f396c060bb0 con 0x7f39540a1140 2023-07-20T01:23:49.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:48 smithi136 ceph-mon[98083]: osd.2 [v2:172.21.15.183:6800/4233496322,v1:172.21.15.183:6801/4233496322] boot 2023-07-20T01:23:49.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:48 smithi136 ceph-mon[98083]: osdmap e18: 3 total, 3 up, 3 in 2023-07-20T01:23:49.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:48 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:23:49.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:48 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-20T01:23:49.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:48 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:23:49.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:48 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:23:49.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:48 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:49.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:48 smithi060 ceph-mon[96524]: osd.2 [v2:172.21.15.183:6800/4233496322,v1:172.21.15.183:6801/4233496322] boot 2023-07-20T01:23:49.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:48 smithi060 ceph-mon[96524]: osdmap e18: 3 total, 3 up, 3 in 2023-07-20T01:23:49.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:48 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:23:49.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:48 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-20T01:23:49.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:48 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:23:49.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:48 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:23:49.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:48 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:49.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:48 smithi183 ceph-mon[98477]: osd.2 [v2:172.21.15.183:6800/4233496322,v1:172.21.15.183:6801/4233496322] boot 2023-07-20T01:23:49.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:48 smithi183 ceph-mon[98477]: osdmap e18: 3 total, 3 up, 3 in 2023-07-20T01:23:49.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:48 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-20T01:23:49.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:48 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-20T01:23:49.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:48 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:23:49.203 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:48 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:23:49.203 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:48 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:49.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:49 smithi060 ceph-mon[96524]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:49.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:49 smithi060 ceph-mon[96524]: from='client.14283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:23:49.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:49 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-20T01:23:49.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:49 smithi060 ceph-mon[96524]: osdmap e19: 3 total, 3 up, 3 in 2023-07-20T01:23:49.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:49 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-20T01:23:50.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:49 smithi136 ceph-mon[98083]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:50.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:49 smithi136 ceph-mon[98083]: from='client.14283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:23:50.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:49 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-20T01:23:50.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:49 smithi136 ceph-mon[98083]: osdmap e19: 3 total, 3 up, 3 in 2023-07-20T01:23:50.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:49 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-20T01:23:50.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:49 smithi183 ceph-mon[98477]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-20T01:23:50.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:49 smithi183 ceph-mon[98477]: from='client.14283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:23:50.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:49 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-20T01:23:50.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:49 smithi183 ceph-mon[98477]: osdmap e19: 3 total, 3 up, 3 in 2023-07-20T01:23:50.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:49 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-20T01:23:50.427 INFO:journalctl@ceph.osd.0.smithi060.stdout:Jul 20 01:23:50 smithi060 sudo[112562]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T01:23:50.427 INFO:journalctl@ceph.osd.0.smithi060.stdout:Jul 20 01:23:50 smithi060 sudo[112562]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T01:23:50.427 INFO:journalctl@ceph.osd.0.smithi060.stdout:Jul 20 01:23:50 smithi060 sudo[112562]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T01:23:50.427 INFO:journalctl@ceph.osd.0.smithi060.stdout:Jul 20 01:23:50 smithi060 sudo[112562]: pam_unix(sudo:session): session closed for user root 2023-07-20T01:23:50.780 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:23:50.797+0000 7f3968ff9700 1 -- 172.21.15.60:0/289143518 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f396400cd00 con 0x7f396c106590 2023-07-20T01:23:51.043 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:50 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-20T01:23:51.043 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:50 smithi136 ceph-mon[98083]: osdmap e20: 3 total, 3 up, 3 in 2023-07-20T01:23:51.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:50 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-20T01:23:51.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:50 smithi060 ceph-mon[96524]: osdmap e20: 3 total, 3 up, 3 in 2023-07-20T01:23:51.178 INFO:journalctl@ceph.osd.0.smithi060.stdout:Jul 20 01:23:50 smithi060 sudo[112600]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T01:23:51.178 INFO:journalctl@ceph.osd.0.smithi060.stdout:Jul 20 01:23:50 smithi060 sudo[112600]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T01:23:51.178 INFO:journalctl@ceph.osd.0.smithi060.stdout:Jul 20 01:23:50 smithi060 sudo[112600]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T01:23:51.178 INFO:journalctl@ceph.osd.0.smithi060.stdout:Jul 20 01:23:50 smithi060 sudo[112600]: pam_unix(sudo:session): session closed for user root 2023-07-20T01:23:51.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:50 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-20T01:23:51.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:50 smithi183 ceph-mon[98477]: osdmap e20: 3 total, 3 up, 3 in 2023-07-20T01:23:51.297 INFO:journalctl@ceph.osd.1.smithi136.stdout:Jul 20 01:23:51 smithi136 sudo[105366]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T01:23:51.297 INFO:journalctl@ceph.osd.1.smithi136.stdout:Jul 20 01:23:51 smithi136 sudo[105366]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T01:23:51.298 INFO:journalctl@ceph.osd.1.smithi136.stdout:Jul 20 01:23:51 smithi136 sudo[105366]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T01:23:51.298 INFO:journalctl@ceph.osd.1.smithi136.stdout:Jul 20 01:23:51 smithi136 sudo[105366]: pam_unix(sudo:session): session closed for user root 2023-07-20T01:23:51.647 INFO:journalctl@ceph.osd.1.smithi136.stdout:Jul 20 01:23:51 smithi136 sudo[105369]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T01:23:51.648 INFO:journalctl@ceph.osd.1.smithi136.stdout:Jul 20 01:23:51 smithi136 sudo[105369]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T01:23:51.649 INFO:journalctl@ceph.osd.1.smithi136.stdout:Jul 20 01:23:51 smithi136 sudo[105369]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T01:23:51.649 INFO:journalctl@ceph.osd.1.smithi136.stdout:Jul 20 01:23:51 smithi136 sudo[105369]: pam_unix(sudo:session): session closed for user root 2023-07-20T01:23:51.820 INFO:journalctl@ceph.osd.2.smithi183.stdout:Jul 20 01:23:51 smithi183 sudo[106495]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T01:23:51.821 INFO:journalctl@ceph.osd.2.smithi183.stdout:Jul 20 01:23:51 smithi183 sudo[106495]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T01:23:51.821 INFO:journalctl@ceph.osd.2.smithi183.stdout:Jul 20 01:23:51 smithi183 sudo[106495]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T01:23:51.821 INFO:journalctl@ceph.osd.2.smithi183.stdout:Jul 20 01:23:51 smithi183 sudo[106495]: pam_unix(sudo:session): session closed for user root 2023-07-20T01:23:51.822 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:51 smithi183 ceph-mon[98477]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T01:23:51.822 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:51 smithi183 ceph-mon[98477]: osdmap e21: 3 total, 3 up, 3 in 2023-07-20T01:23:51.822 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:51 smithi183 ceph-mon[98477]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-20T01:23:51.823 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:51 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:51.823 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:51 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/3593272373' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb99ac8e-19f3-4e76-907f-d1739cbddac2"}]: dispatch 2023-07-20T01:23:51.823 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:51 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/3593272373' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bb99ac8e-19f3-4e76-907f-d1739cbddac2"}]': finished 2023-07-20T01:23:51.823 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:51 smithi183 ceph-mon[98477]: osdmap e22: 4 total, 3 up, 4 in 2023-07-20T01:23:51.823 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:51 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T01:23:51.823 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:51 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:51.823 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:51 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/692353846' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:23:52.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:51 smithi136 ceph-mon[98083]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T01:23:52.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:51 smithi136 ceph-mon[98083]: osdmap e21: 3 total, 3 up, 3 in 2023-07-20T01:23:52.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:51 smithi136 ceph-mon[98083]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-20T01:23:52.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:51 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:52.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:51 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/3593272373' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb99ac8e-19f3-4e76-907f-d1739cbddac2"}]: dispatch 2023-07-20T01:23:52.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:51 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/3593272373' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bb99ac8e-19f3-4e76-907f-d1739cbddac2"}]': finished 2023-07-20T01:23:52.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:51 smithi136 ceph-mon[98083]: osdmap e22: 4 total, 3 up, 4 in 2023-07-20T01:23:52.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:51 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T01:23:52.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:51 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:52.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:51 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/692353846' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:23:52.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:51 smithi060 ceph-mon[96524]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T01:23:52.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:51 smithi060 ceph-mon[96524]: osdmap e21: 3 total, 3 up, 3 in 2023-07-20T01:23:52.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:51 smithi060 ceph-mon[96524]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-20T01:23:52.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:51 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:52.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:51 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/3593272373' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb99ac8e-19f3-4e76-907f-d1739cbddac2"}]: dispatch 2023-07-20T01:23:52.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:51 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/3593272373' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bb99ac8e-19f3-4e76-907f-d1739cbddac2"}]': finished 2023-07-20T01:23:52.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:51 smithi060 ceph-mon[96524]: osdmap e22: 4 total, 3 up, 4 in 2023-07-20T01:23:52.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:51 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T01:23:52.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:51 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:52.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:51 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/692353846' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:23:52.200 INFO:journalctl@ceph.osd.2.smithi183.stdout:Jul 20 01:23:51 smithi183 sudo[106498]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T01:23:52.200 INFO:journalctl@ceph.osd.2.smithi183.stdout:Jul 20 01:23:51 smithi183 sudo[106498]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T01:23:52.200 INFO:journalctl@ceph.osd.2.smithi183.stdout:Jul 20 01:23:51 smithi183 sudo[106498]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T01:23:52.200 INFO:journalctl@ceph.osd.2.smithi183.stdout:Jul 20 01:23:51 smithi183 sudo[106498]: pam_unix(sudo:session): session closed for user root 2023-07-20T01:23:52.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:52 smithi060 sudo[112874]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T01:23:52.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:52 smithi060 sudo[112874]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T01:23:52.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:52 smithi060 sudo[112874]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T01:23:52.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:52 smithi060 sudo[112874]: pam_unix(sudo:session): session closed for user root 2023-07-20T01:23:53.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:52 smithi060 sudo[113063]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T01:23:53.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:52 smithi060 sudo[113063]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T01:23:53.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:52 smithi060 sudo[113063]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T01:23:53.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:52 smithi060 sudo[113063]: pam_unix(sudo:session): session closed for user root 2023-07-20T01:23:53.231 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:53 smithi183 sudo[106501]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T01:23:53.232 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:53 smithi183 sudo[106501]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T01:23:53.232 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:53 smithi183 sudo[106501]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T01:23:53.232 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:53 smithi183 sudo[106501]: pam_unix(sudo:session): session closed for user root 2023-07-20T01:23:53.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:53 smithi183 sudo[106504]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T01:23:53.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:53 smithi183 sudo[106504]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T01:23:53.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:53 smithi183 sudo[106504]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T01:23:53.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:53 smithi183 sudo[106504]: pam_unix(sudo:session): session closed for user root 2023-07-20T01:23:53.727 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:53 smithi136 sudo[105372]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-20T01:23:53.727 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:53 smithi136 sudo[105372]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T01:23:53.727 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:53 smithi136 sudo[105372]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T01:23:53.728 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:53 smithi136 sudo[105372]: pam_unix(sudo:session): session closed for user root 2023-07-20T01:23:54.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:53 smithi136 sudo[105375]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-20T01:23:54.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:53 smithi136 sudo[105375]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-20T01:23:54.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:53 smithi136 sudo[105375]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-20T01:23:54.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:53 smithi136 sudo[105375]: pam_unix(sudo:session): session closed for user root 2023-07-20T01:23:54.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:53 smithi136 ceph-mon[98083]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T01:23:54.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:53 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:54.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:53 smithi136 ceph-mon[98083]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T01:23:54.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:53 smithi136 ceph-mon[98083]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T01:23:54.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:53 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:23:54.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:53 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:23:54.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:53 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:23:54.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:53 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:23:54.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:53 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:23:54.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:53 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:23:54.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:53 smithi060 ceph-mon[96524]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T01:23:54.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:53 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:54.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:53 smithi060 ceph-mon[96524]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T01:23:54.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:53 smithi060 ceph-mon[96524]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T01:23:54.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:53 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:23:54.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:53 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:23:54.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:53 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:23:54.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:53 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:23:54.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:53 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:23:54.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:53 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:23:54.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:53 smithi183 ceph-mon[98477]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T01:23:54.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:53 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:23:54.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:53 smithi183 ceph-mon[98477]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T01:23:54.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:53 smithi183 ceph-mon[98477]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T01:23:54.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:53 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:23:54.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:53 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:23:54.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:53 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:23:54.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:53 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:23:54.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:53 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:23:54.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:53 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:23:54.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:54 smithi060 ceph-mon[96524]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T01:23:54.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:54 smithi060 ceph-mon[96524]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T01:23:54.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:54 smithi060 ceph-mon[96524]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T01:23:54.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:54 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:23:54.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:54 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:23:54.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:54 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:23:54.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:54 smithi060 ceph-mon[96524]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T01:23:55.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:54 smithi136 ceph-mon[98083]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T01:23:55.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:54 smithi136 ceph-mon[98083]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T01:23:55.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:54 smithi136 ceph-mon[98083]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T01:23:55.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:54 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:23:55.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:54 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:23:55.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:54 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:23:55.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:54 smithi136 ceph-mon[98083]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T01:23:55.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:54 smithi183 ceph-mon[98477]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T01:23:55.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:54 smithi183 ceph-mon[98477]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T01:23:55.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:54 smithi183 ceph-mon[98477]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-20T01:23:55.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:54 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-20T01:23:55.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:54 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-20T01:23:55.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:54 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-20T01:23:55.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:54 smithi183 ceph-mon[98477]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-20T01:23:56.005 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:55 smithi060 ceph-mon[96524]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T01:23:56.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:55 smithi136 ceph-mon[98083]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T01:23:56.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:55 smithi183 ceph-mon[98477]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T01:23:57.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:57 smithi060 ceph-mon[96524]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T01:23:57.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:57 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-20T01:23:57.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:57 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:58.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:57 smithi136 ceph-mon[98083]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T01:23:58.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:57 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-20T01:23:58.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:57 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:58.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:57 smithi183 ceph-mon[98477]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T01:23:58.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:57 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-20T01:23:58.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:57 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:23:58.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:58 smithi060 ceph-mon[96524]: Deploying daemon osd.3 on smithi060 2023-07-20T01:23:59.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:58 smithi136 ceph-mon[98083]: Deploying daemon osd.3 on smithi060 2023-07-20T01:23:59.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:58 smithi183 ceph-mon[98477]: Deploying daemon osd.3 on smithi060 2023-07-20T01:24:00.064 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:23:59 smithi060 ceph-mon[96524]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 218 KiB/s wr, 15 op/s 2023-07-20T01:24:00.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:23:59 smithi136 ceph-mon[98083]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 218 KiB/s wr, 15 op/s 2023-07-20T01:24:00.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:23:59 smithi183 ceph-mon[98477]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 218 KiB/s wr, 15 op/s 2023-07-20T01:24:02.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:01 smithi136 ceph-mon[98083]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 163 KiB/s wr, 11 op/s 2023-07-20T01:24:02.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:01 smithi060 ceph-mon[96524]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 163 KiB/s wr, 11 op/s 2023-07-20T01:24:02.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:01 smithi183 ceph-mon[98477]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 163 KiB/s wr, 11 op/s 2023-07-20T01:24:03.142 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:02 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:24:03.142 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:02 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:03.142 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:02 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:03.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:02 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:24:03.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:02 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:03.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:02 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:03.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:02 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:24:03.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:02 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:03.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:02 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:04.075 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:03 smithi060 ceph-mon[96524]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-07-20T01:24:04.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:03 smithi136 ceph-mon[98083]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-07-20T01:24:04.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:03 smithi183 ceph-mon[98477]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-07-20T01:24:05.029 INFO:teuthology.orchestra.run.smithi060.stdout:Created osd(s) 3 on host 'smithi060' 2023-07-20T01:24:05.030 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:05.049+0000 7f3968ff9700 1 -- 172.21.15.60:0/289143518 <== mgr.14152 v2:172.21.15.60:6800/1961833180 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f396c060bb0 con 0x7f39540a1140 2023-07-20T01:24:05.031 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:05.051+0000 7f3972249700 1 -- 172.21.15.60:0/289143518 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f39540a1140 msgr2=0x7f39540a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:24:05.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:05.051+0000 7f3972249700 1 --2- 172.21.15.60:0/289143518 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f39540a1140 0x7f39540a3600 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f395c009c60 tx=0x7f395c009260 comp rx=0 tx=0).stop 2023-07-20T01:24:05.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:05.051+0000 7f3972249700 1 -- 172.21.15.60:0/289143518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f396c106590 msgr2=0x7f396c19b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:24:05.032 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:05.051+0000 7f3972249700 1 --2- 172.21.15.60:0/289143518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f396c106590 0x7f396c19b7c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f396400efc0 tx=0x7f396400c3a0 comp rx=0 tx=0).stop 2023-07-20T01:24:05.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:05.051+0000 7f3972249700 1 -- 172.21.15.60:0/289143518 shutdown_connections 2023-07-20T01:24:05.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:05.051+0000 7f3972249700 1 --2- 172.21.15.60:0/289143518 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f39540a1140 0x7f39540a3600 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:24:05.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:05.051+0000 7f3972249700 1 --2- 172.21.15.60:0/289143518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f396c106590 0x7f396c19b7c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:24:05.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:05.051+0000 7f3972249700 1 --2- 172.21.15.60:0/289143518 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f396c101330 0x7f396c194200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:24:05.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:05.051+0000 7f3972249700 1 --2- 172.21.15.60:0/289143518 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f396c103c60 0x7f396c194740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:24:05.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:05.051+0000 7f3972249700 1 -- 172.21.15.60:0/289143518 >> 172.21.15.60:0/289143518 conn(0x7f396c0fac80 msgr2=0x7f396c101fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:24:05.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:05.051+0000 7f3972249700 1 -- 172.21.15.60:0/289143518 shutdown_connections 2023-07-20T01:24:05.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:05.051+0000 7f3972249700 1 -- 172.21.15.60:0/289143518 wait complete. 2023-07-20T01:24:05.391 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:05 smithi060 ceph-mon[96524]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-07-20T01:24:05.391 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:05 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:05.391 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:05 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:05.392 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:05 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:05.392 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:05 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:24:05.392 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:05 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:05.392 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:05 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:05.392 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:05 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:05.392 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:05 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:24:05.405 DEBUG:teuthology.orchestra.run.smithi060:osd.3> sudo journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.3.service 2023-07-20T01:24:05.408 INFO:tasks.cephadm:Deploying osd.4 on smithi136 with /dev/vg_nvme/lv_3... 2023-07-20T01:24:05.408 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-20T01:24:05.437 INFO:journalctl@ceph.osd.3.smithi060.stdout:-- Logs begin at Thu 2023-07-20 01:00:02 UTC. -- 2023-07-20T01:24:05.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:05 smithi136 ceph-mon[98083]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-07-20T01:24:05.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:05 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:05.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:05 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:05.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:05 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:05.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:05 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:24:05.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:05 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:05.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:05 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:05.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:05 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:05.649 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:05 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:24:05.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:05 smithi183 ceph-mon[98477]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-07-20T01:24:05.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:05 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:05.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:05 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:05.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:05 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:05.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:05 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:24:05.702 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:05 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:05.702 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:05 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:05.702 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:05 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:05.702 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:05 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:24:05.970 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T01:24:06.177 INFO:journalctl@ceph.osd.3.smithi060.stdout:Jul 20 01:24:05 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-3[114879]: 2023-07-20T01:24:05.981+0000 7f717e81a640 -1 osd.3 0 log_to_monitors true 2023-07-20T01:24:06.475 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:06 smithi136 ceph-mon[98083]: from='osd.3 [v2:172.21.15.60:6810/3285594248,v1:172.21.15.60:6811/3285594248]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-20T01:24:06.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:06 smithi060 ceph-mon[96524]: from='osd.3 [v2:172.21.15.60:6810/3285594248,v1:172.21.15.60:6811/3285594248]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-20T01:24:06.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:06 smithi183 ceph-mon[98477]: from='osd.3 [v2:172.21.15.60:6810/3285594248,v1:172.21.15.60:6811/3285594248]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-20T01:24:07.455 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-20T01:24:07.455 INFO:teuthology.orchestra.run.smithi136.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-20T01:24:07.456 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10+0 records in 2023-07-20T01:24:07.456 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2023-07-20T01:24:07.456 INFO:teuthology.orchestra.run.smithi136.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00802243 s, 1.3 GB/s 2023-07-20T01:24:07.456 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping successful for: 2023-07-20T01:24:07.591 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:07 smithi136 ceph-mon[98083]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-07-20T01:24:07.592 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:07 smithi136 ceph-mon[98083]: from='osd.3 [v2:172.21.15.60:6810/3285594248,v1:172.21.15.60:6811/3285594248]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-20T01:24:07.592 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:07 smithi136 ceph-mon[98083]: osdmap e23: 4 total, 3 up, 4 in 2023-07-20T01:24:07.592 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:07 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T01:24:07.592 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:07 smithi136 ceph-mon[98083]: from='osd.3 [v2:172.21.15.60:6810/3285594248,v1:172.21.15.60:6811/3285594248]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2023-07-20T01:24:07.645 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:07 smithi060 ceph-mon[96524]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-07-20T01:24:07.645 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:07 smithi060 ceph-mon[96524]: from='osd.3 [v2:172.21.15.60:6810/3285594248,v1:172.21.15.60:6811/3285594248]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-20T01:24:07.645 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:07 smithi060 ceph-mon[96524]: osdmap e23: 4 total, 3 up, 4 in 2023-07-20T01:24:07.645 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:07 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T01:24:07.645 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:07 smithi060 ceph-mon[96524]: from='osd.3 [v2:172.21.15.60:6810/3285594248,v1:172.21.15.60:6811/3285594248]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2023-07-20T01:24:07.646 INFO:journalctl@ceph.osd.3.smithi060.stdout:Jul 20 01:24:07 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-3[114879]: 2023-07-20T01:24:07.281+0000 7f7173e92700 -1 osd.3 0 waiting for initial osdmap 2023-07-20T01:24:07.646 INFO:journalctl@ceph.osd.3.smithi060.stdout:Jul 20 01:24:07 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-3[114879]: 2023-07-20T01:24:07.286+0000 7f716fc88700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T01:24:07.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:07 smithi183 ceph-mon[98477]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-07-20T01:24:07.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:07 smithi183 ceph-mon[98477]: from='osd.3 [v2:172.21.15.60:6810/3285594248,v1:172.21.15.60:6811/3285594248]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-20T01:24:07.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:07 smithi183 ceph-mon[98477]: osdmap e23: 4 total, 3 up, 4 in 2023-07-20T01:24:07.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:07 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T01:24:07.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:07 smithi183 ceph-mon[98477]: from='osd.3 [v2:172.21.15.60:6810/3285594248,v1:172.21.15.60:6811/3285594248]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2023-07-20T01:24:07.927 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi136:vg_nvme/lv_3 2023-07-20T01:24:08.476 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T01:24:08.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:08 smithi136 ceph-mon[98083]: from='osd.3 [v2:172.21.15.60:6810/3285594248,v1:172.21.15.60:6811/3285594248]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2023-07-20T01:24:08.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:08 smithi136 ceph-mon[98083]: osdmap e24: 4 total, 3 up, 4 in 2023-07-20T01:24:08.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:08 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T01:24:08.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:08 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T01:24:08.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:08 smithi136 ceph-mon[98083]: Detected new or changed devices on smithi060 2023-07-20T01:24:08.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:08 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:08.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:08 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:08.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:08 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:24:08.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:08 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:24:08.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:08 smithi136 ceph-mon[98083]: Adjusting osd_memory_target on smithi060 to 8586M 2023-07-20T01:24:08.649 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:08 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:08.649 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:08 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:08.649 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:08 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:24:08.649 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:08 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:08.649 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:08 smithi136 ceph-mon[98083]: osd.3 [v2:172.21.15.60:6810/3285594248,v1:172.21.15.60:6811/3285594248] boot 2023-07-20T01:24:08.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:08 smithi060 ceph-mon[96524]: from='osd.3 [v2:172.21.15.60:6810/3285594248,v1:172.21.15.60:6811/3285594248]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2023-07-20T01:24:08.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:08 smithi060 ceph-mon[96524]: osdmap e24: 4 total, 3 up, 4 in 2023-07-20T01:24:08.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:08 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T01:24:08.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:08 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T01:24:08.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:08 smithi060 ceph-mon[96524]: Detected new or changed devices on smithi060 2023-07-20T01:24:08.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:08 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:08.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:08 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:08.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:08 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:24:08.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:08 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:24:08.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:08 smithi060 ceph-mon[96524]: Adjusting osd_memory_target on smithi060 to 8586M 2023-07-20T01:24:08.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:08 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:08.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:08 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:08.681 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:08 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:24:08.681 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:08 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:08.681 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:08 smithi060 ceph-mon[96524]: osd.3 [v2:172.21.15.60:6810/3285594248,v1:172.21.15.60:6811/3285594248] boot 2023-07-20T01:24:08.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:08 smithi183 ceph-mon[98477]: from='osd.3 [v2:172.21.15.60:6810/3285594248,v1:172.21.15.60:6811/3285594248]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2023-07-20T01:24:08.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:08 smithi183 ceph-mon[98477]: osdmap e24: 4 total, 3 up, 4 in 2023-07-20T01:24:08.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:08 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T01:24:08.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:08 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T01:24:08.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:08 smithi183 ceph-mon[98477]: Detected new or changed devices on smithi060 2023-07-20T01:24:08.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:08 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:08.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:08 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:08.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:08 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:24:08.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:08 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:24:08.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:08 smithi183 ceph-mon[98477]: Adjusting osd_memory_target on smithi060 to 8586M 2023-07-20T01:24:08.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:08 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:08.702 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:08 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:08.702 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:08 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:24:08.702 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:08 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:08.702 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:08 smithi183 ceph-mon[98477]: osd.3 [v2:172.21.15.60:6810/3285594248,v1:172.21.15.60:6811/3285594248] boot 2023-07-20T01:24:09.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:09 smithi136 ceph-mon[98083]: purged_snaps scrub starts 2023-07-20T01:24:09.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:09 smithi136 ceph-mon[98083]: purged_snaps scrub ok 2023-07-20T01:24:09.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:09 smithi136 ceph-mon[98083]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T01:24:09.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:09 smithi136 ceph-mon[98083]: osdmap e25: 4 total, 4 up, 4 in 2023-07-20T01:24:09.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:09 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T01:24:09.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:09 smithi060 ceph-mon[96524]: purged_snaps scrub starts 2023-07-20T01:24:09.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:09 smithi060 ceph-mon[96524]: purged_snaps scrub ok 2023-07-20T01:24:09.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:09 smithi060 ceph-mon[96524]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T01:24:09.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:09 smithi060 ceph-mon[96524]: osdmap e25: 4 total, 4 up, 4 in 2023-07-20T01:24:09.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:09 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T01:24:09.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:09 smithi183 ceph-mon[98477]: purged_snaps scrub starts 2023-07-20T01:24:09.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:09 smithi183 ceph-mon[98477]: purged_snaps scrub ok 2023-07-20T01:24:09.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:09 smithi183 ceph-mon[98477]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-20T01:24:09.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:09 smithi183 ceph-mon[98477]: osdmap e25: 4 total, 4 up, 4 in 2023-07-20T01:24:09.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:09 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-20T01:24:09.780 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.798+0000 7f00c86b2700 1 -- 172.21.15.136:0/2641515859 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00c01012a0 msgr2=0x7f00c0107b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:24:09.780 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.798+0000 7f00c86b2700 1 --2- 172.21.15.136:0/2641515859 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00c01012a0 0x7f00c0107b50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f00b800b0d0 tx=0x7f00b800b3e0 comp rx=0 tx=0).stop 2023-07-20T01:24:09.780 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.798+0000 7f00c86b2700 1 -- 172.21.15.136:0/2641515859 shutdown_connections 2023-07-20T01:24:09.780 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.798+0000 7f00c86b2700 1 --2- 172.21.15.136:0/2641515859 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00c01012a0 0x7f00c0107b50 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:24:09.780 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.798+0000 7f00c86b2700 1 --2- 172.21.15.136:0/2641515859 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00c00ff700 0x7f00c00ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:24:09.781 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.798+0000 7f00c86b2700 1 --2- 172.21.15.136:0/2641515859 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f00c0100900 0x7f00c0100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:24:09.781 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.798+0000 7f00c86b2700 1 -- 172.21.15.136:0/2641515859 >> 172.21.15.136:0/2641515859 conn(0x7f00c00fac80 msgr2=0x7f00c00fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:24:09.781 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.798+0000 7f00c86b2700 1 -- 172.21.15.136:0/2641515859 shutdown_connections 2023-07-20T01:24:09.781 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.798+0000 7f00c86b2700 1 -- 172.21.15.136:0/2641515859 wait complete. 2023-07-20T01:24:09.781 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.799+0000 7f00c86b2700 1 Processor -- start 2023-07-20T01:24:09.781 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.799+0000 7f00c86b2700 1 -- start start 2023-07-20T01:24:09.781 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.799+0000 7f00c86b2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00c00ff700 0x7f00c01953e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:24:09.781 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.799+0000 7f00c86b2700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00c0100900 0x7f00c0195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:24:09.782 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.799+0000 7f00c86b2700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f00c01012a0 0x7f00c019c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:24:09.782 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.799+0000 7f00c86b2700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c019cf70 con 0x7f00c00ff700 2023-07-20T01:24:09.782 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.799+0000 7f00c86b2700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c019d0b0 con 0x7f00c0100900 2023-07-20T01:24:09.782 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.799+0000 7f00c86b2700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c019d240 con 0x7f00c01012a0 2023-07-20T01:24:09.782 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.799+0000 7f00c5c4d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00c0100900 0x7f00c0195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:24:09.782 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.799+0000 7f00c5c4d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00c0100900 0x7f00c0195920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38020/0 (socket says 172.21.15.136:38020) 2023-07-20T01:24:09.782 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.799+0000 7f00c5c4d700 1 -- 172.21.15.136:0/454781883 learned_addr learned my addr 172.21.15.136:0/454781883 (peer_addr_for_me v2:172.21.15.136:0/0) 2023-07-20T01:24:09.783 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.799+0000 7f00c644e700 1 --2- 172.21.15.136:0/454781883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00c00ff700 0x7f00c01953e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:24:09.783 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.799+0000 7f00c6c4f700 1 --2- 172.21.15.136:0/454781883 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f00c01012a0 0x7f00c019c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:24:09.783 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.799+0000 7f00c5c4d700 1 -- 172.21.15.136:0/454781883 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f00c01012a0 msgr2=0x7f00c019c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:24:09.783 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.799+0000 7f00c5c4d700 1 --2- 172.21.15.136:0/454781883 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f00c01012a0 0x7f00c019c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:24:09.783 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.799+0000 7f00c5c4d700 1 -- 172.21.15.136:0/454781883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00c00ff700 msgr2=0x7f00c01953e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:24:09.783 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.799+0000 7f00c5c4d700 1 --2- 172.21.15.136:0/454781883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00c00ff700 0x7f00c01953e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:24:09.783 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.799+0000 7f00c5c4d700 1 -- 172.21.15.136:0/454781883 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b8009d00 con 0x7f00c0100900 2023-07-20T01:24:09.783 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.799+0000 7f00c6c4f700 1 --2- 172.21.15.136:0/454781883 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f00c01012a0 0x7f00c019c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T01:24:09.784 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.799+0000 7f00c5c4d700 1 --2- 172.21.15.136:0/454781883 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00c0100900 0x7f00c0195920 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f00bc009970 tx=0x7f00bc009c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:24:09.784 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.799+0000 7f00b77fe700 1 -- 172.21.15.136:0/454781883 <== mon.2 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00bc012070 con 0x7f00c0100900 2023-07-20T01:24:09.784 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.800+0000 7f00b77fe700 1 -- 172.21.15.136:0/454781883 <== mon.2 v2:172.21.15.136:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f00bc00d260 con 0x7f00c0100900 2023-07-20T01:24:09.784 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.800+0000 7f00c86b2700 1 -- 172.21.15.136:0/454781883 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00c0068290 con 0x7f00c0100900 2023-07-20T01:24:09.785 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.800+0000 7f00b77fe700 1 -- 172.21.15.136:0/454781883 <== mon.2 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00bc017930 con 0x7f00c0100900 2023-07-20T01:24:09.785 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.800+0000 7f00c86b2700 1 -- 172.21.15.136:0/454781883 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00c00687e0 con 0x7f00c0100900 2023-07-20T01:24:09.785 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.800+0000 7f00c86b2700 1 -- 172.21.15.136:0/454781883 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00c010adc0 con 0x7f00c0100900 2023-07-20T01:24:09.785 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.801+0000 7f00b77fe700 1 -- 172.21.15.136:0/454781883 <== mon.2 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f00bc0076b0 con 0x7f00c0100900 2023-07-20T01:24:09.786 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.802+0000 7f00b77fe700 1 --2- 172.21.15.136:0/454781883 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f00ac0a11d0 0x7f00ac0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:24:09.786 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.802+0000 7f00b77fe700 1 -- 172.21.15.136:0/454781883 <== mon.2 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 3698+0+0 (secure 0 0 0) 0x7f00bc0cc9e0 con 0x7f00c0100900 2023-07-20T01:24:09.786 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.802+0000 7f00c644e700 1 --2- 172.21.15.136:0/454781883 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f00ac0a11d0 0x7f00ac0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:24:09.786 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.803+0000 7f00c644e700 1 --2- 172.21.15.136:0/454781883 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f00ac0a11d0 0x7f00ac0a3690 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f00c0100760 tx=0x7f00b000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:24:09.786 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.806+0000 7f00b77fe700 1 -- 172.21.15.136:0/454781883 <== mon.2 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f00bc097330 con 0x7f00c0100900 2023-07-20T01:24:09.979 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:09.998+0000 7f00c86b2700 1 -- 172.21.15.136:0/454781883 --> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f00c0068a60 con 0x7f00ac0a11d0 2023-07-20T01:24:10.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:10 smithi136 ceph-mon[98083]: osdmap e26: 4 total, 4 up, 4 in 2023-07-20T01:24:10.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:10 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:24:10.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:10 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:24:10.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:10 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:10.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:10 smithi060 ceph-mon[96524]: osdmap e26: 4 total, 4 up, 4 in 2023-07-20T01:24:10.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:10 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:24:10.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:10 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:24:10.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:10 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:10.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:10 smithi183 ceph-mon[98477]: osdmap e26: 4 total, 4 up, 4 in 2023-07-20T01:24:10.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:10 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:24:10.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:10 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:24:10.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:10 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:11.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:11 smithi136 ceph-mon[98083]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 106 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:24:11.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:11 smithi136 ceph-mon[98083]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:24:11.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:11 smithi136 ceph-mon[98083]: osdmap e27: 4 total, 4 up, 4 in 2023-07-20T01:24:11.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:11 smithi060 ceph-mon[96524]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 106 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:24:11.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:11 smithi060 ceph-mon[96524]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:24:11.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:11 smithi060 ceph-mon[96524]: osdmap e27: 4 total, 4 up, 4 in 2023-07-20T01:24:11.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:11 smithi183 ceph-mon[98477]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 106 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:24:11.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:11 smithi183 ceph-mon[98477]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:24:11.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:11 smithi183 ceph-mon[98477]: osdmap e27: 4 total, 4 up, 4 in 2023-07-20T01:24:12.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:12 smithi136 ceph-mon[98083]: osdmap e28: 4 total, 4 up, 4 in 2023-07-20T01:24:12.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:12 smithi060 ceph-mon[96524]: osdmap e28: 4 total, 4 up, 4 in 2023-07-20T01:24:12.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:12 smithi183 ceph-mon[98477]: osdmap e28: 4 total, 4 up, 4 in 2023-07-20T01:24:13.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:13 smithi136 ceph-mon[98083]: pgmap v95: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T01:24:13.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:13 smithi136 ceph-mon[98083]: osdmap e29: 4 total, 4 up, 4 in 2023-07-20T01:24:13.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:13 smithi136 ceph-mon[98083]: from='client.? 172.21.15.136:0/2941127593' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a4e21c8-0151-445f-a94d-852a80b1bce6"}]: dispatch 2023-07-20T01:24:13.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:13 smithi136 ceph-mon[98083]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a4e21c8-0151-445f-a94d-852a80b1bce6"}]: dispatch 2023-07-20T01:24:13.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:13 smithi136 ceph-mon[98083]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a4e21c8-0151-445f-a94d-852a80b1bce6"}]': finished 2023-07-20T01:24:13.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:13 smithi136 ceph-mon[98083]: osdmap e30: 5 total, 4 up, 5 in 2023-07-20T01:24:13.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:13 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T01:24:13.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:13 smithi136 ceph-mon[98083]: from='client.? 172.21.15.136:0/2892488846' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:24:13.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:13 smithi060 ceph-mon[96524]: pgmap v95: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T01:24:13.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:13 smithi060 ceph-mon[96524]: osdmap e29: 4 total, 4 up, 4 in 2023-07-20T01:24:13.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:13 smithi060 ceph-mon[96524]: from='client.? 172.21.15.136:0/2941127593' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a4e21c8-0151-445f-a94d-852a80b1bce6"}]: dispatch 2023-07-20T01:24:13.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:13 smithi060 ceph-mon[96524]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a4e21c8-0151-445f-a94d-852a80b1bce6"}]: dispatch 2023-07-20T01:24:13.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:13 smithi060 ceph-mon[96524]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a4e21c8-0151-445f-a94d-852a80b1bce6"}]': finished 2023-07-20T01:24:13.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:13 smithi060 ceph-mon[96524]: osdmap e30: 5 total, 4 up, 5 in 2023-07-20T01:24:13.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:13 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T01:24:13.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:13 smithi060 ceph-mon[96524]: from='client.? 172.21.15.136:0/2892488846' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:24:13.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:13 smithi183 ceph-mon[98477]: pgmap v95: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T01:24:13.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:13 smithi183 ceph-mon[98477]: osdmap e29: 4 total, 4 up, 4 in 2023-07-20T01:24:13.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:13 smithi183 ceph-mon[98477]: from='client.? 172.21.15.136:0/2941127593' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a4e21c8-0151-445f-a94d-852a80b1bce6"}]: dispatch 2023-07-20T01:24:13.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:13 smithi183 ceph-mon[98477]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a4e21c8-0151-445f-a94d-852a80b1bce6"}]: dispatch 2023-07-20T01:24:13.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:13 smithi183 ceph-mon[98477]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a4e21c8-0151-445f-a94d-852a80b1bce6"}]': finished 2023-07-20T01:24:13.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:13 smithi183 ceph-mon[98477]: osdmap e30: 5 total, 4 up, 5 in 2023-07-20T01:24:13.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:13 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T01:24:13.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:13 smithi183 ceph-mon[98477]: from='client.? 172.21.15.136:0/2892488846' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:24:15.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:15 smithi136 ceph-mon[98083]: pgmap v98: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T01:24:15.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:15 smithi060 ceph-mon[96524]: pgmap v98: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T01:24:15.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:15 smithi183 ceph-mon[98477]: pgmap v98: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-20T01:24:17.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:17 smithi136 ceph-mon[98083]: pgmap v99: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 104 KiB/s, 0 objects/s recovering 2023-07-20T01:24:17.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:17 smithi060 ceph-mon[96524]: pgmap v99: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 104 KiB/s, 0 objects/s recovering 2023-07-20T01:24:17.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:17 smithi183 ceph-mon[98477]: pgmap v99: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 104 KiB/s, 0 objects/s recovering 2023-07-20T01:24:19.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:19 smithi136 ceph-mon[98083]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:24:19.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:19 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-20T01:24:19.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:19 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:19.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:19 smithi060 ceph-mon[96524]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:24:19.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:19 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-20T01:24:19.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:19 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:19.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:19 smithi183 ceph-mon[98477]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:24:19.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:19 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-20T01:24:19.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:19 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:20.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:20 smithi136 ceph-mon[98083]: Deploying daemon osd.4 on smithi136 2023-07-20T01:24:20.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:20 smithi060 ceph-mon[96524]: Deploying daemon osd.4 on smithi136 2023-07-20T01:24:20.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:20 smithi183 ceph-mon[98477]: Deploying daemon osd.4 on smithi136 2023-07-20T01:24:21.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:21 smithi136 ceph-mon[98083]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:24:21.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:21 smithi060 ceph-mon[96524]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:24:21.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:21 smithi183 ceph-mon[98477]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:24:23.646 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:23 smithi136 ceph-mon[98083]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:24:23.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:23 smithi060 ceph-mon[96524]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:24:23.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:23 smithi183 ceph-mon[98477]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:24:24.989 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:24 smithi136 ceph-mon[98083]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:24:24.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:24 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:24.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:24 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:24.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:24 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:24:25.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:24 smithi060 ceph-mon[96524]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:24:25.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:24 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:25.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:24 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:25.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:24 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:24:25.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:24 smithi183 ceph-mon[98477]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:24:25.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:24 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:25.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:24 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:25.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:24 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:24:26.797 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:26.817+0000 7f00b77fe700 1 -- 172.21.15.136:0/454781883 <== mgr.14152 v2:172.21.15.60:6800/1961833180 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f00c0068a60 con 0x7f00ac0a11d0 2023-07-20T01:24:26.798 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 4 on host 'smithi136' 2023-07-20T01:24:26.799 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:26.819+0000 7f00c86b2700 1 -- 172.21.15.136:0/454781883 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f00ac0a11d0 msgr2=0x7f00ac0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:24:26.800 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:26.819+0000 7f00c86b2700 1 --2- 172.21.15.136:0/454781883 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f00ac0a11d0 0x7f00ac0a3690 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f00c0100760 tx=0x7f00b000a380 comp rx=0 tx=0).stop 2023-07-20T01:24:26.800 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:26.819+0000 7f00c86b2700 1 -- 172.21.15.136:0/454781883 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00c0100900 msgr2=0x7f00c0195920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:24:26.800 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:26.819+0000 7f00c86b2700 1 --2- 172.21.15.136:0/454781883 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00c0100900 0x7f00c0195920 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f00bc009970 tx=0x7f00bc009c80 comp rx=0 tx=0).stop 2023-07-20T01:24:26.800 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:26.819+0000 7f00c86b2700 1 -- 172.21.15.136:0/454781883 shutdown_connections 2023-07-20T01:24:26.800 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:26.819+0000 7f00c86b2700 1 --2- 172.21.15.136:0/454781883 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f00ac0a11d0 0x7f00ac0a3690 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:24:26.800 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:26.819+0000 7f00c86b2700 1 --2- 172.21.15.136:0/454781883 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f00c01012a0 0x7f00c019c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:24:26.800 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:26.819+0000 7f00c86b2700 1 --2- 172.21.15.136:0/454781883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00c00ff700 0x7f00c01953e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:24:26.801 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:26.819+0000 7f00c86b2700 1 --2- 172.21.15.136:0/454781883 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00c0100900 0x7f00c0195920 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:24:26.801 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:26.819+0000 7f00c86b2700 1 -- 172.21.15.136:0/454781883 >> 172.21.15.136:0/454781883 conn(0x7f00c00fac80 msgr2=0x7f00c0103e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:24:26.801 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:26.819+0000 7f00c86b2700 1 -- 172.21.15.136:0/454781883 shutdown_connections 2023-07-20T01:24:26.801 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:24:26.820+0000 7f00c86b2700 1 -- 172.21.15.136:0/454781883 wait complete. 2023-07-20T01:24:27.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:27 smithi136 ceph-mon[98083]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:24:27.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:27 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:27.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:27 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:27.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:27 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:27.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:27 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:24:27.150 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:27 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:27.150 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:27 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:27.150 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:27 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:27.150 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:27 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:24:27.267 DEBUG:teuthology.orchestra.run.smithi136:osd.4> sudo journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.4.service 2023-07-20T01:24:27.269 INFO:tasks.cephadm:Deploying osd.5 on smithi183 with /dev/vg_nvme/lv_3... 2023-07-20T01:24:27.269 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-20T01:24:27.300 INFO:journalctl@ceph.osd.4.smithi136.stdout:-- Logs begin at Thu 2023-07-20 01:00:24 UTC. -- 2023-07-20T01:24:27.302 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:27 smithi183 ceph-mon[98477]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:24:27.302 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:27 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:27.302 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:27 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:27.303 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:27 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:27.303 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:27 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:24:27.303 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:27 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:27.303 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:27 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:27.303 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:27 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:27.303 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:27 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:24:27.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:27 smithi060 ceph-mon[96524]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:24:27.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:27 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:27.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:27 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:27.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:27 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:27.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:27 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:24:27.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:27 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:27.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:27 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:27.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:27 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:27.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:27 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:24:27.831 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T01:24:28.042 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jul 20 01:24:27 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-4[108572]: 2023-07-20T01:24:27.850+0000 7f8828b27640 -1 osd.4 0 log_to_monitors true 2023-07-20T01:24:28.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:28 smithi183 ceph-mon[98477]: from='osd.4 [v2:172.21.15.136:6808/2380508636,v1:172.21.15.136:6809/2380508636]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T01:24:28.388 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:28 smithi183 ceph-mon[98477]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T01:24:28.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:28 smithi136 ceph-mon[98083]: from='osd.4 [v2:172.21.15.136:6808/2380508636,v1:172.21.15.136:6809/2380508636]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T01:24:28.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:28 smithi136 ceph-mon[98083]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T01:24:28.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:28 smithi060 ceph-mon[96524]: from='osd.4 [v2:172.21.15.136:6808/2380508636,v1:172.21.15.136:6809/2380508636]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T01:24:28.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:28 smithi060 ceph-mon[96524]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-20T01:24:29.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:29 smithi136 ceph-mon[98083]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:24:29.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:29 smithi136 ceph-mon[98083]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-20T01:24:29.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:29 smithi136 ceph-mon[98083]: osdmap e31: 5 total, 4 up, 5 in 2023-07-20T01:24:29.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:29 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T01:24:29.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:29 smithi136 ceph-mon[98083]: from='osd.4 [v2:172.21.15.136:6808/2380508636,v1:172.21.15.136:6809/2380508636]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-07-20T01:24:29.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:29 smithi136 ceph-mon[98083]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-07-20T01:24:29.148 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jul 20 01:24:29 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-4[108572]: 2023-07-20T01:24:29.069+0000 7f881e19f700 -1 osd.4 0 waiting for initial osdmap 2023-07-20T01:24:29.148 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jul 20 01:24:29 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-4[108572]: 2023-07-20T01:24:29.076+0000 7f8817790700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T01:24:29.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:29 smithi183 ceph-mon[98477]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:24:29.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:29 smithi183 ceph-mon[98477]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-20T01:24:29.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:29 smithi183 ceph-mon[98477]: osdmap e31: 5 total, 4 up, 5 in 2023-07-20T01:24:29.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:29 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T01:24:29.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:29 smithi183 ceph-mon[98477]: from='osd.4 [v2:172.21.15.136:6808/2380508636,v1:172.21.15.136:6809/2380508636]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-07-20T01:24:29.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:29 smithi183 ceph-mon[98477]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-07-20T01:24:29.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:29 smithi060 ceph-mon[96524]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:24:29.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:29 smithi060 ceph-mon[96524]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-20T01:24:29.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:29 smithi060 ceph-mon[96524]: osdmap e31: 5 total, 4 up, 5 in 2023-07-20T01:24:29.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:29 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T01:24:29.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:29 smithi060 ceph-mon[96524]: from='osd.4 [v2:172.21.15.136:6808/2380508636,v1:172.21.15.136:6809/2380508636]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-07-20T01:24:29.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:29 smithi060 ceph-mon[96524]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-07-20T01:24:29.444 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-20T01:24:29.445 INFO:teuthology.orchestra.run.smithi183.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-20T01:24:29.445 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10+0 records in 2023-07-20T01:24:29.445 INFO:teuthology.orchestra.run.smithi183.stderr:10+0 records out 2023-07-20T01:24:29.445 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111962 s, 937 MB/s 2023-07-20T01:24:29.445 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping successful for: 2023-07-20T01:24:29.786 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi183:vg_nvme/lv_3 2023-07-20T01:24:30.380 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T01:24:30.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:30 smithi136 ceph-mon[98083]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2023-07-20T01:24:30.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:30 smithi136 ceph-mon[98083]: osdmap e32: 5 total, 4 up, 5 in 2023-07-20T01:24:30.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:30 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T01:24:30.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:30 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T01:24:30.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:30 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:30.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:30 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:30.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:30 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:24:30.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:30 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:24:30.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:30 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:30.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:30 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:30.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:30 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:24:30.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:30 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:30.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:30 smithi060 ceph-mon[96524]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2023-07-20T01:24:30.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:30 smithi060 ceph-mon[96524]: osdmap e32: 5 total, 4 up, 5 in 2023-07-20T01:24:30.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:30 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T01:24:30.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:30 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T01:24:30.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:30 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:30.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:30 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:30.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:30 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:24:30.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:30 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:24:30.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:30 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:30.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:30 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:30.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:30 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:24:30.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:30 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:30.449 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:30 smithi183 ceph-mon[98477]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2023-07-20T01:24:30.450 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:30 smithi183 ceph-mon[98477]: osdmap e32: 5 total, 4 up, 5 in 2023-07-20T01:24:30.450 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:30 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T01:24:30.450 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:30 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T01:24:30.450 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:30 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:30.450 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:30 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:30.450 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:30 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:24:30.450 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:30 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:24:30.451 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:30 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:30.451 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:30 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:30.451 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:30 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:24:30.451 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:30 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:31.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:31 smithi183 ceph-mon[98477]: purged_snaps scrub starts 2023-07-20T01:24:31.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:31 smithi183 ceph-mon[98477]: purged_snaps scrub ok 2023-07-20T01:24:31.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:31 smithi183 ceph-mon[98477]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:24:31.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:31 smithi183 ceph-mon[98477]: Detected new or changed devices on smithi136 2023-07-20T01:24:31.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:31 smithi183 ceph-mon[98477]: Adjusting osd_memory_target on smithi136 to 8584M 2023-07-20T01:24:31.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:31 smithi183 ceph-mon[98477]: osd.4 [v2:172.21.15.136:6808/2380508636,v1:172.21.15.136:6809/2380508636] boot 2023-07-20T01:24:31.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:31 smithi183 ceph-mon[98477]: osdmap e33: 5 total, 5 up, 5 in 2023-07-20T01:24:31.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:31 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T01:24:31.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:31 smithi136 ceph-mon[98083]: purged_snaps scrub starts 2023-07-20T01:24:31.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:31 smithi136 ceph-mon[98083]: purged_snaps scrub ok 2023-07-20T01:24:31.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:31 smithi136 ceph-mon[98083]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:24:31.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:31 smithi136 ceph-mon[98083]: Detected new or changed devices on smithi136 2023-07-20T01:24:31.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:31 smithi136 ceph-mon[98083]: Adjusting osd_memory_target on smithi136 to 8584M 2023-07-20T01:24:31.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:31 smithi136 ceph-mon[98083]: osd.4 [v2:172.21.15.136:6808/2380508636,v1:172.21.15.136:6809/2380508636] boot 2023-07-20T01:24:31.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:31 smithi136 ceph-mon[98083]: osdmap e33: 5 total, 5 up, 5 in 2023-07-20T01:24:31.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:31 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T01:24:31.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:31 smithi060 ceph-mon[96524]: purged_snaps scrub starts 2023-07-20T01:24:31.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:31 smithi060 ceph-mon[96524]: purged_snaps scrub ok 2023-07-20T01:24:31.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:31 smithi060 ceph-mon[96524]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-20T01:24:31.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:31 smithi060 ceph-mon[96524]: Detected new or changed devices on smithi136 2023-07-20T01:24:31.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:31 smithi060 ceph-mon[96524]: Adjusting osd_memory_target on smithi136 to 8584M 2023-07-20T01:24:31.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:31 smithi060 ceph-mon[96524]: osd.4 [v2:172.21.15.136:6808/2380508636,v1:172.21.15.136:6809/2380508636] boot 2023-07-20T01:24:31.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:31 smithi060 ceph-mon[96524]: osdmap e33: 5 total, 5 up, 5 in 2023-07-20T01:24:31.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:31 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-20T01:24:31.592 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.611+0000 7f0a25035700 1 -- 172.21.15.183:0/89579763 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a200ff700 msgr2=0x7f0a200ffb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:24:31.593 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.611+0000 7f0a25035700 1 --2- 172.21.15.183:0/89579763 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a200ff700 0x7f0a200ffb20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0a14009990 tx=0x7f0a14009ca0 comp rx=0 tx=0).stop 2023-07-20T01:24:31.593 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.611+0000 7f0a25035700 1 -- 172.21.15.183:0/89579763 shutdown_connections 2023-07-20T01:24:31.593 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.611+0000 7f0a25035700 1 --2- 172.21.15.183:0/89579763 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a201012a0 0x7f0a20107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:24:31.593 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.611+0000 7f0a25035700 1 --2- 172.21.15.183:0/89579763 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a200ff700 0x7f0a200ffb20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:24:31.593 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.611+0000 7f0a25035700 1 --2- 172.21.15.183:0/89579763 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a20100900 0x7f0a20100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:24:31.594 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.611+0000 7f0a25035700 1 -- 172.21.15.183:0/89579763 >> 172.21.15.183:0/89579763 conn(0x7f0a200fac80 msgr2=0x7f0a200fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:24:31.594 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.611+0000 7f0a25035700 1 -- 172.21.15.183:0/89579763 shutdown_connections 2023-07-20T01:24:31.594 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.611+0000 7f0a25035700 1 -- 172.21.15.183:0/89579763 wait complete. 2023-07-20T01:24:31.594 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.612+0000 7f0a25035700 1 Processor -- start 2023-07-20T01:24:31.594 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.612+0000 7f0a25035700 1 -- start start 2023-07-20T01:24:31.594 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.612+0000 7f0a25035700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a200ff700 0x7f0a201963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:24:31.594 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.612+0000 7f0a25035700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a20100900 0x7f0a20196900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:24:31.595 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.612+0000 7f0a25035700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a201012a0 0x7f0a2019d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:24:31.595 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.612+0000 7f0a25035700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2019df50 con 0x7f0a201012a0 2023-07-20T01:24:31.595 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.612+0000 7f0a25035700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2019e090 con 0x7f0a200ff700 2023-07-20T01:24:31.595 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.612+0000 7f0a25035700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2019e220 con 0x7f0a20100900 2023-07-20T01:24:31.595 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.612+0000 7f0a1e59c700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a20100900 0x7f0a20196900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:24:31.595 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.612+0000 7f0a1e59c700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a20100900 0x7f0a20196900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:45946/0 (socket says 172.21.15.183:45946) 2023-07-20T01:24:31.596 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.612+0000 7f0a1e59c700 1 -- 172.21.15.183:0/1427955144 learned_addr learned my addr 172.21.15.183:0/1427955144 (peer_addr_for_me v2:172.21.15.183:0/0) 2023-07-20T01:24:31.596 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.612+0000 7f0a1ed9d700 1 --2- 172.21.15.183:0/1427955144 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a200ff700 0x7f0a201963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:24:31.596 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.612+0000 7f0a1f59e700 1 --2- 172.21.15.183:0/1427955144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a201012a0 0x7f0a2019d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:24:31.596 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.612+0000 7f0a1e59c700 1 -- 172.21.15.183:0/1427955144 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a200ff700 msgr2=0x7f0a201963c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:24:31.596 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.612+0000 7f0a1e59c700 1 --2- 172.21.15.183:0/1427955144 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a200ff700 0x7f0a201963c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:24:31.596 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.612+0000 7f0a1e59c700 1 -- 172.21.15.183:0/1427955144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a201012a0 msgr2=0x7f0a2019d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:24:31.596 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.612+0000 7f0a1e59c700 1 --2- 172.21.15.183:0/1427955144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a201012a0 0x7f0a2019d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:24:31.597 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.612+0000 7f0a1e59c700 1 -- 172.21.15.183:0/1427955144 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a140095f0 con 0x7f0a20100900 2023-07-20T01:24:31.597 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.612+0000 7f0a1f59e700 1 --2- 172.21.15.183:0/1427955144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a201012a0 0x7f0a2019d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T01:24:31.597 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.613+0000 7f0a1e59c700 1 --2- 172.21.15.183:0/1427955144 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a20100900 0x7f0a20196900 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0a1000d980 tx=0x7f0a1000dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:24:31.597 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.613+0000 7f0a07fff700 1 -- 172.21.15.183:0/1427955144 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a10009b40 con 0x7f0a20100900 2023-07-20T01:24:31.597 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.613+0000 7f0a25035700 1 -- 172.21.15.183:0/1427955144 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a2019e500 con 0x7f0a20100900 2023-07-20T01:24:31.598 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.613+0000 7f0a07fff700 1 -- 172.21.15.183:0/1427955144 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0a1000b6e0 con 0x7f0a20100900 2023-07-20T01:24:31.598 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.613+0000 7f0a07fff700 1 -- 172.21.15.183:0/1427955144 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a1000f3c0 con 0x7f0a20100900 2023-07-20T01:24:31.599 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.613+0000 7f0a25035700 1 -- 172.21.15.183:0/1427955144 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a2019ea00 con 0x7f0a20100900 2023-07-20T01:24:31.599 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.614+0000 7f0a25035700 1 -- 172.21.15.183:0/1427955144 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a201078c0 con 0x7f0a20100900 2023-07-20T01:24:31.599 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.615+0000 7f0a07fff700 1 -- 172.21.15.183:0/1427955144 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0a10010460 con 0x7f0a20100900 2023-07-20T01:24:31.599 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.616+0000 7f0a07fff700 1 --2- 172.21.15.183:0/1427955144 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f0a080a1150 0x7f0a080a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:24:31.599 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.616+0000 7f0a07fff700 1 -- 172.21.15.183:0/1427955144 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f0a100cb750 con 0x7f0a20100900 2023-07-20T01:24:31.599 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.616+0000 7f0a1ed9d700 1 --2- 172.21.15.183:0/1427955144 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f0a080a1150 0x7f0a080a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:24:31.599 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.616+0000 7f0a1ed9d700 1 --2- 172.21.15.183:0/1427955144 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f0a080a1150 0x7f0a080a3610 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0a14006010 tx=0x7f0a14005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:24:31.600 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.620+0000 7f0a07fff700 1 -- 172.21.15.183:0/1427955144 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0a10095f10 con 0x7f0a20100900 2023-07-20T01:24:31.792 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:31.813+0000 7f0a25035700 1 -- 172.21.15.183:0/1427955144 --> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a20060bb0 con 0x7f0a080a1150 2023-07-20T01:24:32.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:32 smithi136 ceph-mon[98083]: osdmap e34: 5 total, 5 up, 5 in 2023-07-20T01:24:32.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:32 smithi136 ceph-mon[98083]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:24:32.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:32 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:24:32.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:32 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:24:32.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:32 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:32.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:32 smithi060 ceph-mon[96524]: osdmap e34: 5 total, 5 up, 5 in 2023-07-20T01:24:32.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:32 smithi060 ceph-mon[96524]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:24:32.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:32 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:24:32.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:32 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:24:32.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:32 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:32.449 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:32 smithi183 ceph-mon[98477]: osdmap e34: 5 total, 5 up, 5 in 2023-07-20T01:24:32.450 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:32 smithi183 ceph-mon[98477]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:24:32.450 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:32 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:24:32.450 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:32 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:24:32.450 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:32 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:33.646 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:33 smithi136 ceph-mon[98083]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:24:33.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:33 smithi060 ceph-mon[96524]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:24:33.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:33 smithi183 ceph-mon[98477]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:24:34.646 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:34 smithi136 ceph-mon[98083]: from='client.? 172.21.15.183:0/3072064509' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9e51980-2c9e-40ea-ad3f-cb233018455f"}]: dispatch 2023-07-20T01:24:34.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:34 smithi136 ceph-mon[98083]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9e51980-2c9e-40ea-ad3f-cb233018455f"}]: dispatch 2023-07-20T01:24:34.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:34 smithi136 ceph-mon[98083]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a9e51980-2c9e-40ea-ad3f-cb233018455f"}]': finished 2023-07-20T01:24:34.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:34 smithi136 ceph-mon[98083]: osdmap e35: 6 total, 5 up, 6 in 2023-07-20T01:24:34.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:34 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T01:24:34.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:34 smithi060 ceph-mon[96524]: from='client.? 172.21.15.183:0/3072064509' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9e51980-2c9e-40ea-ad3f-cb233018455f"}]: dispatch 2023-07-20T01:24:34.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:34 smithi060 ceph-mon[96524]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9e51980-2c9e-40ea-ad3f-cb233018455f"}]: dispatch 2023-07-20T01:24:34.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:34 smithi060 ceph-mon[96524]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a9e51980-2c9e-40ea-ad3f-cb233018455f"}]': finished 2023-07-20T01:24:34.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:34 smithi060 ceph-mon[96524]: osdmap e35: 6 total, 5 up, 6 in 2023-07-20T01:24:34.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:34 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T01:24:34.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:34 smithi183 ceph-mon[98477]: from='client.? 172.21.15.183:0/3072064509' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9e51980-2c9e-40ea-ad3f-cb233018455f"}]: dispatch 2023-07-20T01:24:34.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:34 smithi183 ceph-mon[98477]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9e51980-2c9e-40ea-ad3f-cb233018455f"}]: dispatch 2023-07-20T01:24:34.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:34 smithi183 ceph-mon[98477]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a9e51980-2c9e-40ea-ad3f-cb233018455f"}]': finished 2023-07-20T01:24:34.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:34 smithi183 ceph-mon[98477]: osdmap e35: 6 total, 5 up, 6 in 2023-07-20T01:24:34.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:34 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T01:24:35.646 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:35 smithi136 ceph-mon[98083]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:24:35.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:35 smithi136 ceph-mon[98083]: from='client.? 172.21.15.183:0/2715607649' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:24:35.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:35 smithi060 ceph-mon[96524]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:24:35.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:35 smithi060 ceph-mon[96524]: from='client.? 172.21.15.183:0/2715607649' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:24:35.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:35 smithi183 ceph-mon[98477]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:24:35.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:35 smithi183 ceph-mon[98477]: from='client.? 172.21.15.183:0/2715607649' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:24:37.646 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:37 smithi136 ceph-mon[98083]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:24:37.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:37 smithi060 ceph-mon[96524]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:24:37.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:37 smithi183 ceph-mon[98477]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:24:39.646 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:39 smithi136 ceph-mon[98083]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:24:39.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:39 smithi060 ceph-mon[96524]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:24:39.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:39 smithi183 ceph-mon[98477]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:24:41.626 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:41 smithi183 ceph-mon[98477]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:24:41.626 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:41 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-20T01:24:41.627 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:41 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:41.627 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:41 smithi183 ceph-mon[98477]: Deploying daemon osd.5 on smithi183 2023-07-20T01:24:41.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:41 smithi136 ceph-mon[98083]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:24:41.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:41 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-20T01:24:41.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:41 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:41.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:41 smithi136 ceph-mon[98083]: Deploying daemon osd.5 on smithi183 2023-07-20T01:24:41.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:41 smithi060 ceph-mon[96524]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:24:41.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:41 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-20T01:24:41.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:41 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:41.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:41 smithi060 ceph-mon[96524]: Deploying daemon osd.5 on smithi183 2023-07-20T01:24:43.450 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:43 smithi183 ceph-mon[98477]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:24:43.646 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:43 smithi136 ceph-mon[98083]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:24:43.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:43 smithi060 ceph-mon[96524]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:24:45.623 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:45 smithi183 ceph-mon[98477]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:24:45.623 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:45 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:24:45.623 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:45 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:45.624 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:45 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:45.646 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:45 smithi136 ceph-mon[98083]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:24:45.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:45 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:24:45.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:45 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:45.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:45 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:45.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:45 smithi060 ceph-mon[96524]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:24:45.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:45 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:24:45.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:45 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:45.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:45 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:47.646 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:47 smithi136 ceph-mon[98083]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:24:47.675 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:47 smithi183 ceph-mon[98477]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:24:47.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:47 smithi060 ceph-mon[96524]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:24:47.963 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:47.983+0000 7f0a07fff700 1 -- 172.21.15.183:0/1427955144 <== mgr.14152 v2:172.21.15.60:6800/1961833180 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0a20060bb0 con 0x7f0a080a1150 2023-07-20T01:24:47.963 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 5 on host 'smithi183' 2023-07-20T01:24:47.965 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:47.986+0000 7f0a25035700 1 -- 172.21.15.183:0/1427955144 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f0a080a1150 msgr2=0x7f0a080a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:24:47.965 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:47.986+0000 7f0a25035700 1 --2- 172.21.15.183:0/1427955144 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f0a080a1150 0x7f0a080a3610 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0a14006010 tx=0x7f0a14005960 comp rx=0 tx=0).stop 2023-07-20T01:24:47.966 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:47.986+0000 7f0a25035700 1 -- 172.21.15.183:0/1427955144 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a20100900 msgr2=0x7f0a20196900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:24:47.966 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:47.986+0000 7f0a25035700 1 --2- 172.21.15.183:0/1427955144 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a20100900 0x7f0a20196900 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0a1000d980 tx=0x7f0a1000dc90 comp rx=0 tx=0).stop 2023-07-20T01:24:47.966 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:47.986+0000 7f0a25035700 1 -- 172.21.15.183:0/1427955144 shutdown_connections 2023-07-20T01:24:47.966 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:47.986+0000 7f0a25035700 1 --2- 172.21.15.183:0/1427955144 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f0a080a1150 0x7f0a080a3610 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:24:47.966 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:47.986+0000 7f0a25035700 1 --2- 172.21.15.183:0/1427955144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a201012a0 0x7f0a2019d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:24:47.966 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:47.986+0000 7f0a25035700 1 --2- 172.21.15.183:0/1427955144 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a200ff700 0x7f0a201963c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:24:47.966 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:47.986+0000 7f0a25035700 1 --2- 172.21.15.183:0/1427955144 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a20100900 0x7f0a20196900 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:24:47.967 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:47.986+0000 7f0a25035700 1 -- 172.21.15.183:0/1427955144 >> 172.21.15.183:0/1427955144 conn(0x7f0a200fac80 msgr2=0x7f0a200fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:24:47.967 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:47.986+0000 7f0a25035700 1 -- 172.21.15.183:0/1427955144 shutdown_connections 2023-07-20T01:24:47.967 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:24:47.986+0000 7f0a25035700 1 -- 172.21.15.183:0/1427955144 wait complete. 2023-07-20T01:24:48.480 DEBUG:teuthology.orchestra.run.smithi183:osd.5> sudo journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.5.service 2023-07-20T01:24:48.483 INFO:tasks.cephadm:Deploying osd.6 on smithi060 with /dev/vg_nvme/lv_2... 2023-07-20T01:24:48.483 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-20T01:24:48.527 INFO:journalctl@ceph.osd.5.smithi183.stdout:-- Logs begin at Thu 2023-07-20 01:00:08 UTC. -- 2023-07-20T01:24:48.856 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:48 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:48.857 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:48 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:48.857 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:48 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:48.857 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:48 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:24:48.857 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:48 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:48.857 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:48 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:48.858 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:48 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:48.858 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:48 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:24:48.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:48 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:48.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:48 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:48.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:48 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:48.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:48 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:24:48.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:48 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:48.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:48 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:48.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:48 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:48.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:48 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:24:49.070 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:24:49.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:48 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:49.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:48 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:49.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:48 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:49.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:48 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:24:49.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:48 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:49.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:48 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:49.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:48 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:49.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:48 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:24:49.450 INFO:journalctl@ceph.osd.5.smithi183.stdout:Jul 20 01:24:49 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-5[109748]: 2023-07-20T01:24:49.201+0000 7f5df45c5640 -1 osd.5 0 log_to_monitors true 2023-07-20T01:24:49.742 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:49 smithi183 ceph-mon[98477]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:24:49.743 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:49 smithi183 ceph-mon[98477]: from='osd.5 [v2:172.21.15.183:6808/2401792837,v1:172.21.15.183:6809/2401792837]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T01:24:49.743 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:49 smithi183 ceph-mon[98477]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T01:24:49.853 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:49 smithi060 ceph-mon[96524]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:24:49.854 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:49 smithi060 ceph-mon[96524]: from='osd.5 [v2:172.21.15.183:6808/2401792837,v1:172.21.15.183:6809/2401792837]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T01:24:49.854 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:49 smithi060 ceph-mon[96524]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T01:24:50.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:49 smithi136 ceph-mon[98083]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:24:50.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:49 smithi136 ceph-mon[98083]: from='osd.5 [v2:172.21.15.183:6808/2401792837,v1:172.21.15.183:6809/2401792837]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T01:24:50.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:49 smithi136 ceph-mon[98083]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-20T01:24:50.809 INFO:teuthology.orchestra.run.smithi060.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-20T01:24:50.809 INFO:teuthology.orchestra.run.smithi060.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-20T01:24:50.809 INFO:teuthology.orchestra.run.smithi060.stderr: stderr: 10+0 records in 2023-07-20T01:24:50.810 INFO:teuthology.orchestra.run.smithi060.stderr:10+0 records out 2023-07-20T01:24:50.810 INFO:teuthology.orchestra.run.smithi060.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0138613 s, 756 MB/s 2023-07-20T01:24:50.810 INFO:teuthology.orchestra.run.smithi060.stderr:--> Zapping successful for: 2023-07-20T01:24:50.949 INFO:journalctl@ceph.osd.5.smithi183.stdout:Jul 20 01:24:50 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-5[109748]: 2023-07-20T01:24:50.739+0000 7f5dea43e700 -1 osd.5 0 waiting for initial osdmap 2023-07-20T01:24:50.949 INFO:journalctl@ceph.osd.5.smithi183.stdout:Jul 20 01:24:50 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-5[109748]: 2023-07-20T01:24:50.747+0000 7f5de3a2f700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T01:24:50.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:50 smithi183 ceph-mon[98477]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-20T01:24:50.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:50 smithi183 ceph-mon[98477]: osdmap e36: 6 total, 5 up, 6 in 2023-07-20T01:24:50.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:50 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T01:24:50.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:50 smithi183 ceph-mon[98477]: from='osd.5 [v2:172.21.15.183:6808/2401792837,v1:172.21.15.183:6809/2401792837]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2023-07-20T01:24:50.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:50 smithi183 ceph-mon[98477]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2023-07-20T01:24:50.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:50 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:50.966 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:50 smithi060 ceph-mon[96524]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-20T01:24:50.967 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:50 smithi060 ceph-mon[96524]: osdmap e36: 6 total, 5 up, 6 in 2023-07-20T01:24:50.967 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:50 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T01:24:50.967 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:50 smithi060 ceph-mon[96524]: from='osd.5 [v2:172.21.15.183:6808/2401792837,v1:172.21.15.183:6809/2401792837]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2023-07-20T01:24:50.967 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:50 smithi060 ceph-mon[96524]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2023-07-20T01:24:50.967 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:50 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:51.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:50 smithi136 ceph-mon[98083]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-20T01:24:51.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:50 smithi136 ceph-mon[98083]: osdmap e36: 6 total, 5 up, 6 in 2023-07-20T01:24:51.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:50 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T01:24:51.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:50 smithi136 ceph-mon[98083]: from='osd.5 [v2:172.21.15.183:6808/2401792837,v1:172.21.15.183:6809/2401792837]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2023-07-20T01:24:51.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:50 smithi136 ceph-mon[98083]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2023-07-20T01:24:51.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:50 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:51.219 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi060:vg_nvme/lv_2 2023-07-20T01:24:51.779 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:24:51.981 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:51 smithi060 ceph-mon[96524]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:24:51.982 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:51 smithi060 ceph-mon[96524]: Detected new or changed devices on smithi183 2023-07-20T01:24:51.982 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:51 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:51.982 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:51 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:24:51.982 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:51 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:24:51.982 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:51 smithi060 ceph-mon[96524]: Adjusting osd_memory_target on smithi183 to 8584M 2023-07-20T01:24:51.983 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:51 smithi060 ceph-mon[96524]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2023-07-20T01:24:51.983 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:51 smithi060 ceph-mon[96524]: osdmap e37: 6 total, 5 up, 6 in 2023-07-20T01:24:51.983 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:51 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T01:24:51.983 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:51 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T01:24:51.983 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:51 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:51.983 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:51 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:51.983 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:51 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:24:51.983 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:51 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:51.984 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:51 smithi060 ceph-mon[96524]: osd.5 [v2:172.21.15.183:6808/2401792837,v1:172.21.15.183:6809/2401792837] boot 2023-07-20T01:24:51.984 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:51 smithi060 ceph-mon[96524]: osdmap e38: 6 total, 6 up, 6 in 2023-07-20T01:24:51.984 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:51 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T01:24:52.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:51 smithi136 ceph-mon[98083]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:24:52.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:51 smithi136 ceph-mon[98083]: Detected new or changed devices on smithi183 2023-07-20T01:24:52.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:51 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:52.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:51 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:24:52.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:51 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:24:52.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:51 smithi136 ceph-mon[98083]: Adjusting osd_memory_target on smithi183 to 8584M 2023-07-20T01:24:52.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:51 smithi136 ceph-mon[98083]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2023-07-20T01:24:52.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:51 smithi136 ceph-mon[98083]: osdmap e37: 6 total, 5 up, 6 in 2023-07-20T01:24:52.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:51 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T01:24:52.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:51 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T01:24:52.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:51 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:52.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:51 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:52.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:51 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:24:52.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:51 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:52.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:51 smithi136 ceph-mon[98083]: osd.5 [v2:172.21.15.183:6808/2401792837,v1:172.21.15.183:6809/2401792837] boot 2023-07-20T01:24:52.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:51 smithi136 ceph-mon[98083]: osdmap e38: 6 total, 6 up, 6 in 2023-07-20T01:24:52.149 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:51 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T01:24:52.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:51 smithi183 ceph-mon[98477]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-20T01:24:52.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:51 smithi183 ceph-mon[98477]: Detected new or changed devices on smithi183 2023-07-20T01:24:52.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:51 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:52.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:51 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:24:52.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:51 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:24:52.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:51 smithi183 ceph-mon[98477]: Adjusting osd_memory_target on smithi183 to 8584M 2023-07-20T01:24:52.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:51 smithi183 ceph-mon[98477]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2023-07-20T01:24:52.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:51 smithi183 ceph-mon[98477]: osdmap e37: 6 total, 5 up, 6 in 2023-07-20T01:24:52.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:51 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T01:24:52.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:51 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T01:24:52.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:51 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:52.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:51 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:52.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:51 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:24:52.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:51 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:24:52.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:51 smithi183 ceph-mon[98477]: osd.5 [v2:172.21.15.183:6808/2401792837,v1:172.21.15.183:6809/2401792837] boot 2023-07-20T01:24:52.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:51 smithi183 ceph-mon[98477]: osdmap e38: 6 total, 6 up, 6 in 2023-07-20T01:24:52.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:51 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-20T01:24:52.788 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:52 smithi060 ceph-mon[96524]: purged_snaps scrub starts 2023-07-20T01:24:52.789 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:52 smithi060 ceph-mon[96524]: purged_snaps scrub ok 2023-07-20T01:24:52.789 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:52 smithi060 ceph-mon[96524]: osdmap e39: 6 total, 6 up, 6 in 2023-07-20T01:24:52.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.995+0000 7f0fa2054700 1 -- 172.21.15.60:0/3702045408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f9c100900 msgr2=0x7f0f9c100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:24:52.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.995+0000 7f0fa2054700 1 --2- 172.21.15.60:0/3702045408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f9c100900 0x7f0f9c100d60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c0099c0 tx=0x7f0f8c009cd0 comp rx=0 tx=0).stop 2023-07-20T01:24:52.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.995+0000 7f0fa2054700 1 -- 172.21.15.60:0/3702045408 shutdown_connections 2023-07-20T01:24:52.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.995+0000 7f0fa2054700 1 --2- 172.21.15.60:0/3702045408 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f9c1012a0 0x7f0f9c107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:24:52.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.995+0000 7f0fa2054700 1 --2- 172.21.15.60:0/3702045408 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f9c0ff700 0x7f0f9c0ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:24:52.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.995+0000 7f0fa2054700 1 --2- 172.21.15.60:0/3702045408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f9c100900 0x7f0f9c100d60 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:24:52.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.995+0000 7f0fa2054700 1 -- 172.21.15.60:0/3702045408 >> 172.21.15.60:0/3702045408 conn(0x7f0f9c0fac80 msgr2=0x7f0f9c0fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:24:52.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.995+0000 7f0fa2054700 1 -- 172.21.15.60:0/3702045408 shutdown_connections 2023-07-20T01:24:52.975 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.995+0000 7f0fa2054700 1 -- 172.21.15.60:0/3702045408 wait complete. 2023-07-20T01:24:52.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.996+0000 7f0fa2054700 1 Processor -- start 2023-07-20T01:24:52.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.996+0000 7f0fa2054700 1 -- start start 2023-07-20T01:24:52.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.996+0000 7f0fa2054700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f9c0ff700 0x7f0f9c068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:24:52.977 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.996+0000 7f0fa2054700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f9c100900 0x7f0f9c19afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:24:52.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.996+0000 7f0fa2054700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f9c1012a0 0x7f0f9c19d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:24:52.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.996+0000 7f0fa2054700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c19ddc0 con 0x7f0f9c0ff700 2023-07-20T01:24:52.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.996+0000 7f0fa2054700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c19df30 con 0x7f0f9c100900 2023-07-20T01:24:52.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.996+0000 7f0fa2054700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c19e0f0 con 0x7f0f9c1012a0 2023-07-20T01:24:52.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.996+0000 7f0f9b7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f9c0ff700 0x7f0f9c068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:24:52.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.996+0000 7f0f9b7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f9c0ff700 0x7f0f9c068db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42418/0 (socket says 172.21.15.60:42418) 2023-07-20T01:24:52.978 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.996+0000 7f0f9b7fe700 1 -- 172.21.15.60:0/4205639304 learned_addr learned my addr 172.21.15.60:0/4205639304 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:24:52.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.997+0000 7f0f9affd700 1 --2- 172.21.15.60:0/4205639304 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f9c100900 0x7f0f9c19afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:24:52.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.997+0000 7f0f9b7fe700 1 -- 172.21.15.60:0/4205639304 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f9c1012a0 msgr2=0x7f0f9c19d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:24:52.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.997+0000 7f0f9b7fe700 1 --2- 172.21.15.60:0/4205639304 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f9c1012a0 0x7f0f9c19d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:24:52.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.997+0000 7f0f9b7fe700 1 -- 172.21.15.60:0/4205639304 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f9c100900 msgr2=0x7f0f9c19afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:24:52.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.997+0000 7f0f9b7fe700 1 --2- 172.21.15.60:0/4205639304 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f9c100900 0x7f0f9c19afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:24:52.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.997+0000 7f0f9b7fe700 1 -- 172.21.15.60:0/4205639304 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f8c0095f0 con 0x7f0f9c0ff700 2023-07-20T01:24:52.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.997+0000 7f0f9b7fe700 1 --2- 172.21.15.60:0/4205639304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f9c0ff700 0x7f0f9c068db0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0f9000c990 tx=0x7f0f9000cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:24:52.979 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.997+0000 7f0f98ff9700 1 -- 172.21.15.60:0/4205639304 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f90007b00 con 0x7f0f9c0ff700 2023-07-20T01:24:52.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.997+0000 7f0fa2054700 1 -- 172.21.15.60:0/4205639304 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f9c19e3d0 con 0x7f0f9c0ff700 2023-07-20T01:24:52.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.997+0000 7f0f98ff9700 1 -- 172.21.15.60:0/4205639304 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0f90007c60 con 0x7f0f9c0ff700 2023-07-20T01:24:52.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.997+0000 7f0f98ff9700 1 -- 172.21.15.60:0/4205639304 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f90018660 con 0x7f0f9c0ff700 2023-07-20T01:24:52.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.997+0000 7f0fa2054700 1 -- 172.21.15.60:0/4205639304 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f9c19e8a0 con 0x7f0f9c0ff700 2023-07-20T01:24:52.980 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.998+0000 7f0fa2054700 1 -- 172.21.15.60:0/4205639304 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f9c106610 con 0x7f0f9c0ff700 2023-07-20T01:24:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:52.999+0000 7f0f98ff9700 1 -- 172.21.15.60:0/4205639304 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0f9000f450 con 0x7f0f9c0ff700 2023-07-20T01:24:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:53.000+0000 7f0f98ff9700 1 --2- 172.21.15.60:0/4205639304 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f0f840a1150 0x7f0f840a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:24:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:53.000+0000 7f0f9affd700 1 --2- 172.21.15.60:0/4205639304 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f0f840a1150 0x7f0f840a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:24:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:53.000+0000 7f0f98ff9700 1 -- 172.21.15.60:0/4205639304 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4562+0+0 (secure 0 0 0) 0x7f0f900cbb40 con 0x7f0f9c0ff700 2023-07-20T01:24:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:53.000+0000 7f0f9affd700 1 --2- 172.21.15.60:0/4205639304 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f0f840a1150 0x7f0f840a3610 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c009990 tx=0x7f0f8c005bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:24:52.982 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:53.004+0000 7f0f98ff9700 1 -- 172.21.15.60:0/4205639304 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0f90096120 con 0x7f0f9c0ff700 2023-07-20T01:24:53.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:52 smithi136 ceph-mon[98083]: purged_snaps scrub starts 2023-07-20T01:24:53.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:52 smithi136 ceph-mon[98083]: purged_snaps scrub ok 2023-07-20T01:24:53.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:52 smithi136 ceph-mon[98083]: osdmap e39: 6 total, 6 up, 6 in 2023-07-20T01:24:53.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:24:53.211+0000 7f0fa2054700 1 -- 172.21.15.60:0/4205639304 --> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f9c060bb0 con 0x7f0f840a1150 2023-07-20T01:24:53.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:52 smithi183 ceph-mon[98477]: purged_snaps scrub starts 2023-07-20T01:24:53.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:52 smithi183 ceph-mon[98477]: purged_snaps scrub ok 2023-07-20T01:24:53.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:52 smithi183 ceph-mon[98477]: osdmap e39: 6 total, 6 up, 6 in 2023-07-20T01:24:53.997 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:53 smithi060 ceph-mon[96524]: pgmap v125: 1 pgs: 1 peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-20T01:24:53.997 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:53 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:24:53.997 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:53 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:24:53.997 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:53 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:53.997 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:53 smithi060 ceph-mon[96524]: osdmap e40: 6 total, 6 up, 6 in 2023-07-20T01:24:54.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:53 smithi136 ceph-mon[98083]: pgmap v125: 1 pgs: 1 peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-20T01:24:54.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:53 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:24:54.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:53 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:24:54.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:53 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:54.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:53 smithi136 ceph-mon[98083]: osdmap e40: 6 total, 6 up, 6 in 2023-07-20T01:24:54.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:53 smithi183 ceph-mon[98477]: pgmap v125: 1 pgs: 1 peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-20T01:24:54.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:53 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:24:54.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:53 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:24:54.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:53 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:24:54.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:53 smithi183 ceph-mon[98477]: osdmap e40: 6 total, 6 up, 6 in 2023-07-20T01:24:55.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:54 smithi136 ceph-mon[98083]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:24:55.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:54 smithi060 ceph-mon[96524]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:24:55.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:54 smithi183 ceph-mon[98477]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:24:56.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:55 smithi136 ceph-mon[98083]: pgmap v128: 1 pgs: 1 peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-20T01:24:56.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:55 smithi136 ceph-mon[98083]: osdmap e41: 6 total, 6 up, 6 in 2023-07-20T01:24:56.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:55 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/2039593022' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58ff5768-d850-40f2-b90c-1098dfdc3256"}]: dispatch 2023-07-20T01:24:56.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:55 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/2039593022' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "58ff5768-d850-40f2-b90c-1098dfdc3256"}]': finished 2023-07-20T01:24:56.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:55 smithi136 ceph-mon[98083]: osdmap e42: 7 total, 6 up, 7 in 2023-07-20T01:24:56.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:55 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:24:56.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:55 smithi060 ceph-mon[96524]: pgmap v128: 1 pgs: 1 peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-20T01:24:56.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:55 smithi060 ceph-mon[96524]: osdmap e41: 6 total, 6 up, 6 in 2023-07-20T01:24:56.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:55 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/2039593022' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58ff5768-d850-40f2-b90c-1098dfdc3256"}]: dispatch 2023-07-20T01:24:56.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:55 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/2039593022' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "58ff5768-d850-40f2-b90c-1098dfdc3256"}]': finished 2023-07-20T01:24:56.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:55 smithi060 ceph-mon[96524]: osdmap e42: 7 total, 6 up, 7 in 2023-07-20T01:24:56.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:55 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:24:56.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:55 smithi183 ceph-mon[98477]: pgmap v128: 1 pgs: 1 peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-20T01:24:56.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:55 smithi183 ceph-mon[98477]: osdmap e41: 6 total, 6 up, 6 in 2023-07-20T01:24:56.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:55 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/2039593022' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58ff5768-d850-40f2-b90c-1098dfdc3256"}]: dispatch 2023-07-20T01:24:56.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:55 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/2039593022' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "58ff5768-d850-40f2-b90c-1098dfdc3256"}]': finished 2023-07-20T01:24:56.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:55 smithi183 ceph-mon[98477]: osdmap e42: 7 total, 6 up, 7 in 2023-07-20T01:24:56.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:55 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:24:57.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:56 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/1790016471' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:24:57.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:56 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/1790016471' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:24:57.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:56 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/1790016471' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:24:58.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:57 smithi136 ceph-mon[98083]: pgmap v131: 1 pgs: 1 peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-20T01:24:58.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:57 smithi060 ceph-mon[96524]: pgmap v131: 1 pgs: 1 peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-20T01:24:58.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:57 smithi183 ceph-mon[98477]: pgmap v131: 1 pgs: 1 peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-20T01:24:59.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:24:59 smithi060 ceph-mon[96524]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 105 KiB/s, 0 objects/s recovering 2023-07-20T01:25:00.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:24:59 smithi136 ceph-mon[98083]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 105 KiB/s, 0 objects/s recovering 2023-07-20T01:25:00.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:24:59 smithi183 ceph-mon[98477]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 105 KiB/s, 0 objects/s recovering 2023-07-20T01:25:02.086 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:01 smithi060 ceph-mon[96524]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2023-07-20T01:25:02.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:01 smithi136 ceph-mon[98083]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2023-07-20T01:25:02.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:01 smithi183 ceph-mon[98477]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2023-07-20T01:25:03.106 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:02 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-20T01:25:03.106 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:02 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:03.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:02 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-20T01:25:03.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:02 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:03.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:02 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-20T01:25:03.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:02 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:03.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:03 smithi060 ceph-mon[96524]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T01:25:03.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:03 smithi060 ceph-mon[96524]: Deploying daemon osd.6 on smithi060 2023-07-20T01:25:04.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:03 smithi136 ceph-mon[98083]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T01:25:04.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:03 smithi136 ceph-mon[98083]: Deploying daemon osd.6 on smithi060 2023-07-20T01:25:04.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:03 smithi183 ceph-mon[98477]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T01:25:04.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:03 smithi183 ceph-mon[98477]: Deploying daemon osd.6 on smithi060 2023-07-20T01:25:05.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:05 smithi060 ceph-mon[96524]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-07-20T01:25:06.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:05 smithi136 ceph-mon[98083]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-07-20T01:25:06.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:05 smithi183 ceph-mon[98477]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-07-20T01:25:07.639 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:07 smithi060 ceph-mon[96524]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 56 KiB/s, 0 objects/s recovering 2023-07-20T01:25:07.640 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:07 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:07.640 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:07 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:25:07.640 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:07 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:07.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:07 smithi136 ceph-mon[98083]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 56 KiB/s, 0 objects/s recovering 2023-07-20T01:25:07.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:07 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:07.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:07 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:25:07.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:07 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:07.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:07 smithi183 ceph-mon[98477]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 56 KiB/s, 0 objects/s recovering 2023-07-20T01:25:07.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:07 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:07.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:07 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:25:07.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:07 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:09.245 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:09.267+0000 7f0f98ff9700 1 -- 172.21.15.60:0/4205639304 <== mgr.14152 v2:172.21.15.60:6800/1961833180 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0f9c060bb0 con 0x7f0f840a1150 2023-07-20T01:25:09.246 INFO:teuthology.orchestra.run.smithi060.stdout:Created osd(s) 6 on host 'smithi060' 2023-07-20T01:25:09.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:09.269+0000 7f0fa2054700 1 -- 172.21.15.60:0/4205639304 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f0f840a1150 msgr2=0x7f0f840a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:25:09.248 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:09.269+0000 7f0fa2054700 1 --2- 172.21.15.60:0/4205639304 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f0f840a1150 0x7f0f840a3610 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c009990 tx=0x7f0f8c005bc0 comp rx=0 tx=0).stop 2023-07-20T01:25:09.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:09.269+0000 7f0fa2054700 1 -- 172.21.15.60:0/4205639304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f9c0ff700 msgr2=0x7f0f9c068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:25:09.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:09.269+0000 7f0fa2054700 1 --2- 172.21.15.60:0/4205639304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f9c0ff700 0x7f0f9c068db0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0f9000c990 tx=0x7f0f9000cca0 comp rx=0 tx=0).stop 2023-07-20T01:25:09.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:09.270+0000 7f0fa2054700 1 -- 172.21.15.60:0/4205639304 shutdown_connections 2023-07-20T01:25:09.249 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:09.270+0000 7f0fa2054700 1 --2- 172.21.15.60:0/4205639304 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f0f840a1150 0x7f0f840a3610 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:25:09.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:09.270+0000 7f0fa2054700 1 --2- 172.21.15.60:0/4205639304 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f9c1012a0 0x7f0f9c19d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:25:09.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:09.270+0000 7f0fa2054700 1 --2- 172.21.15.60:0/4205639304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f9c0ff700 0x7f0f9c068db0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:25:09.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:09.270+0000 7f0fa2054700 1 --2- 172.21.15.60:0/4205639304 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f9c100900 0x7f0f9c19afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:25:09.250 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:09.270+0000 7f0fa2054700 1 -- 172.21.15.60:0/4205639304 >> 172.21.15.60:0/4205639304 conn(0x7f0f9c0fac80 msgr2=0x7f0f9c0fcf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:25:09.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:09.270+0000 7f0fa2054700 1 -- 172.21.15.60:0/4205639304 shutdown_connections 2023-07-20T01:25:09.251 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:09.270+0000 7f0fa2054700 1 -- 172.21.15.60:0/4205639304 wait complete. 2023-07-20T01:25:09.683 DEBUG:teuthology.orchestra.run.smithi060:osd.6> sudo journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.6.service 2023-07-20T01:25:09.686 INFO:tasks.cephadm:Deploying osd.7 on smithi136 with /dev/vg_nvme/lv_2... 2023-07-20T01:25:09.686 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-20T01:25:09.723 INFO:journalctl@ceph.osd.6.smithi060.stdout:-- Logs begin at Thu 2023-07-20 01:00:02 UTC. -- 2023-07-20T01:25:09.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:09 smithi136 ceph-mon[98083]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-20T01:25:09.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:09 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:09.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:09 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:09.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:09 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:09.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:09 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:25:09.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:09 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:09.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:09 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:25:09.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:09 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:09.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:09 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:09.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:09 smithi060 ceph-mon[96524]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-20T01:25:09.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:09 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:09.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:09 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:09.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:09 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:09.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:09 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:25:09.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:09 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:09.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:09 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:25:09.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:09 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:09.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:09 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:09.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:09 smithi183 ceph-mon[98477]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-20T01:25:09.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:09 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:09.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:09 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:09.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:09 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:09.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:09 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:25:09.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:09 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:09.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:09 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:25:09.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:09 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:09.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:09 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:10.259 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T01:25:10.860 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:10 smithi136 ceph-mon[98083]: from='osd.6 [v2:172.21.15.60:6818/3834224571,v1:172.21.15.60:6819/3834224571]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-20T01:25:10.921 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:10 smithi060 ceph-mon[96524]: from='osd.6 [v2:172.21.15.60:6818/3834224571,v1:172.21.15.60:6819/3834224571]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-20T01:25:10.922 INFO:journalctl@ceph.osd.6.smithi060.stdout:Jul 20 01:25:10 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-6[120307]: 2023-07-20T01:25:10.522+0000 7fc531427640 -1 osd.6 0 log_to_monitors true 2023-07-20T01:25:10.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:10 smithi183 ceph-mon[98477]: from='osd.6 [v2:172.21.15.60:6818/3834224571,v1:172.21.15.60:6819/3834224571]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-20T01:25:11.646 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:11 smithi136 ceph-mon[98083]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-20T01:25:11.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:11 smithi136 ceph-mon[98083]: from='osd.6 [v2:172.21.15.60:6818/3834224571,v1:172.21.15.60:6819/3834224571]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-20T01:25:11.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:11 smithi136 ceph-mon[98083]: osdmap e43: 7 total, 6 up, 7 in 2023-07-20T01:25:11.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:11 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:25:11.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:11 smithi136 ceph-mon[98083]: from='osd.6 [v2:172.21.15.60:6818/3834224571,v1:172.21.15.60:6819/3834224571]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2023-07-20T01:25:11.858 INFO:journalctl@ceph.osd.6.smithi060.stdout:Jul 20 01:25:11 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-6[120307]: 2023-07-20T01:25:11.546+0000 7fc526a9f700 -1 osd.6 0 waiting for initial osdmap 2023-07-20T01:25:11.858 INFO:journalctl@ceph.osd.6.smithi060.stdout:Jul 20 01:25:11 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-6[120307]: 2023-07-20T01:25:11.557+0000 7fc522895700 -1 osd.6 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T01:25:11.859 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:11 smithi060 ceph-mon[96524]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-20T01:25:11.859 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:11 smithi060 ceph-mon[96524]: from='osd.6 [v2:172.21.15.60:6818/3834224571,v1:172.21.15.60:6819/3834224571]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-20T01:25:11.859 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:11 smithi060 ceph-mon[96524]: osdmap e43: 7 total, 6 up, 7 in 2023-07-20T01:25:11.859 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:11 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:25:11.860 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:11 smithi060 ceph-mon[96524]: from='osd.6 [v2:172.21.15.60:6818/3834224571,v1:172.21.15.60:6819/3834224571]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2023-07-20T01:25:11.864 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-20T01:25:11.864 INFO:teuthology.orchestra.run.smithi136.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-20T01:25:11.865 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10+0 records in 2023-07-20T01:25:11.865 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2023-07-20T01:25:11.865 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00697951 s, 1.5 GB/s 2023-07-20T01:25:11.866 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping successful for: 2023-07-20T01:25:11.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:11 smithi183 ceph-mon[98477]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-20T01:25:11.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:11 smithi183 ceph-mon[98477]: from='osd.6 [v2:172.21.15.60:6818/3834224571,v1:172.21.15.60:6819/3834224571]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-20T01:25:11.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:11 smithi183 ceph-mon[98477]: osdmap e43: 7 total, 6 up, 7 in 2023-07-20T01:25:11.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:11 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:25:11.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:11 smithi183 ceph-mon[98477]: from='osd.6 [v2:172.21.15.60:6818/3834224571,v1:172.21.15.60:6819/3834224571]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2023-07-20T01:25:12.313 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi136:vg_nvme/lv_2 2023-07-20T01:25:12.864 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T01:25:12.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:12 smithi136 ceph-mon[98083]: from='osd.6 [v2:172.21.15.60:6818/3834224571,v1:172.21.15.60:6819/3834224571]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2023-07-20T01:25:12.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:12 smithi136 ceph-mon[98083]: osdmap e44: 7 total, 6 up, 7 in 2023-07-20T01:25:12.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:12 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:25:12.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:12 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:25:12.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:12 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:12.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:12 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:12.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:12 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:25:12.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:12 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:25:12.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:12 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:25:12.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:12 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:12.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:12 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:12.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:12 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:25:12.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:12 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:12.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:12 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:25:12.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:12 smithi060 ceph-mon[96524]: from='osd.6 [v2:172.21.15.60:6818/3834224571,v1:172.21.15.60:6819/3834224571]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2023-07-20T01:25:12.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:12 smithi060 ceph-mon[96524]: osdmap e44: 7 total, 6 up, 7 in 2023-07-20T01:25:12.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:12 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:25:12.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:12 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:25:12.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:12 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:12.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:12 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:12.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:12 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:25:12.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:12 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:25:12.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:12 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:25:12.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:12 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:12.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:12 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:12.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:12 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:25:12.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:12 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:12.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:12 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:25:12.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:12 smithi183 ceph-mon[98477]: from='osd.6 [v2:172.21.15.60:6818/3834224571,v1:172.21.15.60:6819/3834224571]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2023-07-20T01:25:12.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:12 smithi183 ceph-mon[98477]: osdmap e44: 7 total, 6 up, 7 in 2023-07-20T01:25:12.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:12 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:25:12.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:12 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:25:12.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:12 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:12.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:12 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:12.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:12 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:25:12.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:12 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:25:12.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:12 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:25:12.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:12 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:12.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:12 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:12.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:12 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:25:12.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:12 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:12.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:12 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:25:13.624 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:13 smithi136 ceph-mon[98083]: purged_snaps scrub starts 2023-07-20T01:25:13.625 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:13 smithi136 ceph-mon[98083]: purged_snaps scrub ok 2023-07-20T01:25:13.625 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:13 smithi136 ceph-mon[98083]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-20T01:25:13.625 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:13 smithi136 ceph-mon[98083]: Detected new or changed devices on smithi060 2023-07-20T01:25:13.625 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:13 smithi136 ceph-mon[98083]: Adjusting osd_memory_target on smithi060 to 5724M 2023-07-20T01:25:13.626 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:13 smithi136 ceph-mon[98083]: osd.6 [v2:172.21.15.60:6818/3834224571,v1:172.21.15.60:6819/3834224571] boot 2023-07-20T01:25:13.626 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:13 smithi136 ceph-mon[98083]: osdmap e45: 7 total, 7 up, 7 in 2023-07-20T01:25:13.626 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:13 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:25:13.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:13 smithi060 ceph-mon[96524]: purged_snaps scrub starts 2023-07-20T01:25:13.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:13 smithi060 ceph-mon[96524]: purged_snaps scrub ok 2023-07-20T01:25:13.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:13 smithi060 ceph-mon[96524]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-20T01:25:13.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:13 smithi060 ceph-mon[96524]: Detected new or changed devices on smithi060 2023-07-20T01:25:13.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:13 smithi060 ceph-mon[96524]: Adjusting osd_memory_target on smithi060 to 5724M 2023-07-20T01:25:13.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:13 smithi060 ceph-mon[96524]: osd.6 [v2:172.21.15.60:6818/3834224571,v1:172.21.15.60:6819/3834224571] boot 2023-07-20T01:25:13.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:13 smithi060 ceph-mon[96524]: osdmap e45: 7 total, 7 up, 7 in 2023-07-20T01:25:13.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:13 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:25:13.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:13 smithi183 ceph-mon[98477]: purged_snaps scrub starts 2023-07-20T01:25:13.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:13 smithi183 ceph-mon[98477]: purged_snaps scrub ok 2023-07-20T01:25:13.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:13 smithi183 ceph-mon[98477]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-20T01:25:13.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:13 smithi183 ceph-mon[98477]: Detected new or changed devices on smithi060 2023-07-20T01:25:13.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:13 smithi183 ceph-mon[98477]: Adjusting osd_memory_target on smithi060 to 5724M 2023-07-20T01:25:13.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:13 smithi183 ceph-mon[98477]: osd.6 [v2:172.21.15.60:6818/3834224571,v1:172.21.15.60:6819/3834224571] boot 2023-07-20T01:25:13.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:13 smithi183 ceph-mon[98477]: osdmap e45: 7 total, 7 up, 7 in 2023-07-20T01:25:13.952 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:13 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-20T01:25:14.063 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.083+0000 7f4e3a847700 1 -- 172.21.15.136:0/783762196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e340ff9d0 msgr2=0x7f4e340ffe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:25:14.063 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.083+0000 7f4e3a847700 1 --2- 172.21.15.136:0/783762196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e340ff9d0 0x7f4e340ffe30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4e28009990 tx=0x7f4e28009ca0 comp rx=0 tx=0).stop 2023-07-20T01:25:14.063 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.083+0000 7f4e3a847700 1 -- 172.21.15.136:0/783762196 shutdown_connections 2023-07-20T01:25:14.064 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.084+0000 7f4e3a847700 1 --2- 172.21.15.136:0/783762196 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e34100370 0x7f4e34107c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:25:14.064 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.084+0000 7f4e3a847700 1 --2- 172.21.15.136:0/783762196 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e34100c90 0x7f4e341010b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:25:14.064 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.084+0000 7f4e3a847700 1 --2- 172.21.15.136:0/783762196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e340ff9d0 0x7f4e340ffe30 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:25:14.064 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.084+0000 7f4e3a847700 1 -- 172.21.15.136:0/783762196 >> 172.21.15.136:0/783762196 conn(0x7f4e340fa420 msgr2=0x7f4e340fc860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:25:14.064 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.084+0000 7f4e3a847700 1 -- 172.21.15.136:0/783762196 shutdown_connections 2023-07-20T01:25:14.064 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.084+0000 7f4e3a847700 1 -- 172.21.15.136:0/783762196 wait complete. 2023-07-20T01:25:14.066 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.084+0000 7f4e3a847700 1 Processor -- start 2023-07-20T01:25:14.066 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.084+0000 7f4e3a847700 1 -- start start 2023-07-20T01:25:14.066 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.084+0000 7f4e3a847700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e340ff9d0 0x7f4e340ff480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:25:14.066 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.084+0000 7f4e3a847700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e34100370 0x7f4e3419b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:25:14.067 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.084+0000 7f4e3a847700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e34100c90 0x7f4e3419d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:25:14.067 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.084+0000 7f4e3a847700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3419dea0 con 0x7f4e34100c90 2023-07-20T01:25:14.067 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.084+0000 7f4e3a847700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3419e010 con 0x7f4e34100370 2023-07-20T01:25:14.067 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.084+0000 7f4e3a847700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3419e1d0 con 0x7f4e340ff9d0 2023-07-20T01:25:14.067 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.085+0000 7f4e337fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e34100370 0x7f4e3419b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:25:14.067 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.085+0000 7f4e337fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e34100370 0x7f4e3419b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38134/0 (socket says 172.21.15.136:38134) 2023-07-20T01:25:14.067 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.085+0000 7f4e337fe700 1 -- 172.21.15.136:0/2933571796 learned_addr learned my addr 172.21.15.136:0/2933571796 (peer_addr_for_me v2:172.21.15.136:0/0) 2023-07-20T01:25:14.067 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.085+0000 7f4e33fff700 1 --2- 172.21.15.136:0/2933571796 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e340ff9d0 0x7f4e340ff480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:25:14.068 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.085+0000 7f4e38de4700 1 --2- 172.21.15.136:0/2933571796 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e34100c90 0x7f4e3419d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:25:14.068 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.085+0000 7f4e337fe700 1 -- 172.21.15.136:0/2933571796 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e340ff9d0 msgr2=0x7f4e340ff480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:25:14.068 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.085+0000 7f4e337fe700 1 --2- 172.21.15.136:0/2933571796 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e340ff9d0 0x7f4e340ff480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:25:14.068 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.085+0000 7f4e337fe700 1 -- 172.21.15.136:0/2933571796 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e34100c90 msgr2=0x7f4e3419d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:25:14.068 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.085+0000 7f4e337fe700 1 --2- 172.21.15.136:0/2933571796 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e34100c90 0x7f4e3419d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:25:14.068 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.085+0000 7f4e337fe700 1 -- 172.21.15.136:0/2933571796 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e280095f0 con 0x7f4e34100370 2023-07-20T01:25:14.068 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.085+0000 7f4e337fe700 1 --2- 172.21.15.136:0/2933571796 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e34100370 0x7f4e3419b070 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4e28000c00 tx=0x7f4e28003a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:25:14.069 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.085+0000 7f4e317fa700 1 -- 172.21.15.136:0/2933571796 <== mon.2 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2801d070 con 0x7f4e34100370 2023-07-20T01:25:14.069 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.085+0000 7f4e3a847700 1 -- 172.21.15.136:0/2933571796 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e3419e450 con 0x7f4e34100370 2023-07-20T01:25:14.069 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.085+0000 7f4e317fa700 1 -- 172.21.15.136:0/2933571796 <== mon.2 v2:172.21.15.136:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4e2800bd70 con 0x7f4e34100370 2023-07-20T01:25:14.069 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.085+0000 7f4e317fa700 1 -- 172.21.15.136:0/2933571796 <== mon.2 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e28022c80 con 0x7f4e34100370 2023-07-20T01:25:14.069 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.085+0000 7f4e3a847700 1 -- 172.21.15.136:0/2933571796 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e3419e8c0 con 0x7f4e34100370 2023-07-20T01:25:14.069 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.086+0000 7f4e3a847700 1 -- 172.21.15.136:0/2933571796 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e3410ae00 con 0x7f4e34100370 2023-07-20T01:25:14.070 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.087+0000 7f4e317fa700 1 -- 172.21.15.136:0/2933571796 <== mon.2 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4e28022470 con 0x7f4e34100370 2023-07-20T01:25:14.070 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.088+0000 7f4e317fa700 1 --2- 172.21.15.136:0/2933571796 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f4e1c0a1150 0x7f4e1c0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:25:14.070 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.088+0000 7f4e317fa700 1 -- 172.21.15.136:0/2933571796 <== mon.2 v2:172.21.15.136:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f4e280cd220 con 0x7f4e34100370 2023-07-20T01:25:14.070 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.088+0000 7f4e33fff700 1 --2- 172.21.15.136:0/2933571796 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f4e1c0a1150 0x7f4e1c0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:25:14.071 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.088+0000 7f4e33fff700 1 --2- 172.21.15.136:0/2933571796 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f4e1c0a1150 0x7f4e1c0a3610 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4e34067fb0 tx=0x7f4e2400a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:25:14.071 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.092+0000 7f4e317fa700 1 -- 172.21.15.136:0/2933571796 <== mon.2 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4e28097710 con 0x7f4e34100370 2023-07-20T01:25:14.261 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:14.282+0000 7f4e3a847700 1 -- 172.21.15.136:0/2933571796 --> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e34060bb0 con 0x7f4e1c0a1150 2023-07-20T01:25:14.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:14 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:25:14.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:14 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:25:14.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:14 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:14.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:14 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:25:14.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:14 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:25:14.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:14 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:14.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:14 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:25:14.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:14 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:25:14.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:14 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:15.646 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:15 smithi136 ceph-mon[98083]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:15.646 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:15 smithi136 ceph-mon[98083]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:25:15.646 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:15 smithi136 ceph-mon[98083]: osdmap e46: 7 total, 7 up, 7 in 2023-07-20T01:25:15.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:15 smithi060 ceph-mon[96524]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:15.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:15 smithi060 ceph-mon[96524]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:25:15.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:15 smithi060 ceph-mon[96524]: osdmap e46: 7 total, 7 up, 7 in 2023-07-20T01:25:15.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:15 smithi183 ceph-mon[98477]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:15.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:15 smithi183 ceph-mon[98477]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:25:15.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:15 smithi183 ceph-mon[98477]: osdmap e46: 7 total, 7 up, 7 in 2023-07-20T01:25:17.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:17 smithi136 ceph-mon[98083]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:17.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:17 smithi136 ceph-mon[98083]: from='client.? 172.21.15.136:0/2300064287' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "375cd893-5f26-42bb-98ed-30b05911a7af"}]: dispatch 2023-07-20T01:25:17.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:17 smithi136 ceph-mon[98083]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "375cd893-5f26-42bb-98ed-30b05911a7af"}]: dispatch 2023-07-20T01:25:17.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:17 smithi136 ceph-mon[98083]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "375cd893-5f26-42bb-98ed-30b05911a7af"}]': finished 2023-07-20T01:25:17.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:17 smithi136 ceph-mon[98083]: osdmap e47: 8 total, 7 up, 8 in 2023-07-20T01:25:17.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:17 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:25:17.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:17 smithi136 ceph-mon[98083]: from='client.? 172.21.15.136:0/1482419438' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:25:17.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:17 smithi060 ceph-mon[96524]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:17.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:17 smithi060 ceph-mon[96524]: from='client.? 172.21.15.136:0/2300064287' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "375cd893-5f26-42bb-98ed-30b05911a7af"}]: dispatch 2023-07-20T01:25:17.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:17 smithi060 ceph-mon[96524]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "375cd893-5f26-42bb-98ed-30b05911a7af"}]: dispatch 2023-07-20T01:25:17.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:17 smithi060 ceph-mon[96524]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "375cd893-5f26-42bb-98ed-30b05911a7af"}]': finished 2023-07-20T01:25:17.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:17 smithi060 ceph-mon[96524]: osdmap e47: 8 total, 7 up, 8 in 2023-07-20T01:25:17.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:17 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:25:17.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:17 smithi060 ceph-mon[96524]: from='client.? 172.21.15.136:0/1482419438' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:25:17.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:17 smithi183 ceph-mon[98477]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:17.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:17 smithi183 ceph-mon[98477]: from='client.? 172.21.15.136:0/2300064287' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "375cd893-5f26-42bb-98ed-30b05911a7af"}]: dispatch 2023-07-20T01:25:17.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:17 smithi183 ceph-mon[98477]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "375cd893-5f26-42bb-98ed-30b05911a7af"}]: dispatch 2023-07-20T01:25:17.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:17 smithi183 ceph-mon[98477]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "375cd893-5f26-42bb-98ed-30b05911a7af"}]': finished 2023-07-20T01:25:17.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:17 smithi183 ceph-mon[98477]: osdmap e47: 8 total, 7 up, 8 in 2023-07-20T01:25:17.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:17 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:25:17.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:17 smithi183 ceph-mon[98477]: from='client.? 172.21.15.136:0/1482419438' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:25:19.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:19 smithi136 ceph-mon[98083]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:19.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:19 smithi060 ceph-mon[96524]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:19.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:19 smithi183 ceph-mon[98477]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:21.838 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:21 smithi136 ceph-mon[98083]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:21.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:21 smithi060 ceph-mon[96524]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:21.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:21 smithi183 ceph-mon[98477]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:23.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:23 smithi136 ceph-mon[98083]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:23.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:23 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-20T01:25:23.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:23 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:23.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:23 smithi060 ceph-mon[96524]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:23.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:23 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-20T01:25:23.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:23 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:23.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:23 smithi183 ceph-mon[98477]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:23.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:23 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-20T01:25:23.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:23 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:24.627 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:24 smithi136 ceph-mon[98083]: Deploying daemon osd.7 on smithi136 2023-07-20T01:25:24.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:24 smithi060 ceph-mon[96524]: Deploying daemon osd.7 on smithi136 2023-07-20T01:25:24.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:24 smithi183 ceph-mon[98477]: Deploying daemon osd.7 on smithi136 2023-07-20T01:25:25.816 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:25 smithi136 ceph-mon[98083]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:25.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:25 smithi060 ceph-mon[96524]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:25.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:25 smithi183 ceph-mon[98477]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:27.848 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:27 smithi136 ceph-mon[98083]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:27.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:27 smithi060 ceph-mon[96524]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:27.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:27 smithi183 ceph-mon[98477]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:29.214 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:29 smithi136 ceph-mon[98083]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:29.215 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:29 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:29.215 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:29 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:29.215 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:29 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:25:29.425 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:29 smithi060 ceph-mon[96524]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:29.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:29 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:29.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:29 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:29.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:29 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:25:29.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:29 smithi183 ceph-mon[98477]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:29.449 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:29 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:29.449 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:29 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:29.449 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:29 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:25:31.178 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 7 on host 'smithi136' 2023-07-20T01:25:31.179 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:31.199+0000 7f4e317fa700 1 -- 172.21.15.136:0/2933571796 <== mgr.14152 v2:172.21.15.60:6800/1961833180 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4e34060bb0 con 0x7f4e1c0a1150 2023-07-20T01:25:31.180 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:31.202+0000 7f4e3a847700 1 -- 172.21.15.136:0/2933571796 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f4e1c0a1150 msgr2=0x7f4e1c0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:25:31.180 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:31.202+0000 7f4e3a847700 1 --2- 172.21.15.136:0/2933571796 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f4e1c0a1150 0x7f4e1c0a3610 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4e34067fb0 tx=0x7f4e2400a3b0 comp rx=0 tx=0).stop 2023-07-20T01:25:31.180 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:31.202+0000 7f4e3a847700 1 -- 172.21.15.136:0/2933571796 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e34100370 msgr2=0x7f4e3419b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:25:31.180 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:31.202+0000 7f4e3a847700 1 --2- 172.21.15.136:0/2933571796 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e34100370 0x7f4e3419b070 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4e28000c00 tx=0x7f4e28003a50 comp rx=0 tx=0).stop 2023-07-20T01:25:31.181 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:31.202+0000 7f4e3a847700 1 -- 172.21.15.136:0/2933571796 shutdown_connections 2023-07-20T01:25:31.181 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:31.202+0000 7f4e3a847700 1 --2- 172.21.15.136:0/2933571796 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f4e1c0a1150 0x7f4e1c0a3610 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:25:31.181 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:31.202+0000 7f4e3a847700 1 --2- 172.21.15.136:0/2933571796 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e34100c90 0x7f4e3419d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:25:31.181 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:31.202+0000 7f4e3a847700 1 --2- 172.21.15.136:0/2933571796 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e340ff9d0 0x7f4e340ff480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:25:31.181 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:31.202+0000 7f4e3a847700 1 --2- 172.21.15.136:0/2933571796 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e34100370 0x7f4e3419b070 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:25:31.181 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:31.202+0000 7f4e3a847700 1 -- 172.21.15.136:0/2933571796 >> 172.21.15.136:0/2933571796 conn(0x7f4e340fa420 msgr2=0x7f4e340fc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:25:31.182 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:31.202+0000 7f4e3a847700 1 -- 172.21.15.136:0/2933571796 shutdown_connections 2023-07-20T01:25:31.182 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:25:31.202+0000 7f4e3a847700 1 -- 172.21.15.136:0/2933571796 wait complete. 2023-07-20T01:25:31.646 DEBUG:teuthology.orchestra.run.smithi136:osd.7> sudo journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.7.service 2023-07-20T01:25:31.648 INFO:tasks.cephadm:Deploying osd.8 on smithi183 with /dev/vg_nvme/lv_2... 2023-07-20T01:25:31.648 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-20T01:25:31.679 INFO:journalctl@ceph.osd.7.smithi136.stdout:-- Logs begin at Thu 2023-07-20 01:00:24 UTC. -- 2023-07-20T01:25:31.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:31 smithi136 ceph-mon[98083]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:31.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:31 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:31.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:31 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:31.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:31 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:31.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:31 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:25:31.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:31 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:31.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:31 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:31.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:31 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:31.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:31 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:25:31.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:31 smithi060 ceph-mon[96524]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:31.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:31 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:31.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:31 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:31.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:31 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:31.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:31 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:25:31.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:31 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:31.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:31 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:31.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:31 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:31.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:31 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:25:31.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:31 smithi183 ceph-mon[98477]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:31.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:31 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:31.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:31 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:31.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:31 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:31.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:31 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:25:31.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:31 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:31.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:31 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:31.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:31 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:31.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:31 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:25:32.209 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T01:25:32.237 INFO:journalctl@ceph.osd.7.smithi136.stdout:Jul 20 01:25:32 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-7[113977]: 2023-07-20T01:25:31.999+0000 7f4ac987d640 -1 osd.7 0 log_to_monitors true 2023-07-20T01:25:32.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:32 smithi060 ceph-mon[96524]: from='osd.7 [v2:172.21.15.136:6816/404531062,v1:172.21.15.136:6817/404531062]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T01:25:32.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:32 smithi060 ceph-mon[96524]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T01:25:32.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:32 smithi183 ceph-mon[98477]: from='osd.7 [v2:172.21.15.136:6816/404531062,v1:172.21.15.136:6817/404531062]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T01:25:32.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:32 smithi183 ceph-mon[98477]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T01:25:33.054 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:32 smithi136 ceph-mon[98083]: from='osd.7 [v2:172.21.15.136:6816/404531062,v1:172.21.15.136:6817/404531062]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T01:25:33.054 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:32 smithi136 ceph-mon[98083]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-20T01:25:33.787 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-20T01:25:33.788 INFO:teuthology.orchestra.run.smithi183.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-20T01:25:33.788 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10+0 records in 2023-07-20T01:25:33.788 INFO:teuthology.orchestra.run.smithi183.stderr:10+0 records out 2023-07-20T01:25:33.788 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109469 s, 958 MB/s 2023-07-20T01:25:33.788 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping successful for: 2023-07-20T01:25:33.895 INFO:journalctl@ceph.osd.7.smithi136.stdout:Jul 20 01:25:33 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-7[113977]: 2023-07-20T01:25:33.677+0000 7f4abeef5700 -1 osd.7 0 waiting for initial osdmap 2023-07-20T01:25:33.896 INFO:journalctl@ceph.osd.7.smithi136.stdout:Jul 20 01:25:33 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-7[113977]: 2023-07-20T01:25:33.687+0000 7f4ab8ce7700 -1 osd.7 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T01:25:33.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:33 smithi136 ceph-mon[98083]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:33.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:33 smithi136 ceph-mon[98083]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-20T01:25:33.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:33 smithi136 ceph-mon[98083]: osdmap e48: 8 total, 7 up, 8 in 2023-07-20T01:25:33.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:33 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:25:33.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:33 smithi136 ceph-mon[98083]: from='osd.7 [v2:172.21.15.136:6816/404531062,v1:172.21.15.136:6817/404531062]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-07-20T01:25:33.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:33 smithi136 ceph-mon[98083]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-07-20T01:25:33.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:33 smithi060 ceph-mon[96524]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:33.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:33 smithi060 ceph-mon[96524]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-20T01:25:33.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:33 smithi060 ceph-mon[96524]: osdmap e48: 8 total, 7 up, 8 in 2023-07-20T01:25:33.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:33 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:25:33.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:33 smithi060 ceph-mon[96524]: from='osd.7 [v2:172.21.15.136:6816/404531062,v1:172.21.15.136:6817/404531062]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-07-20T01:25:33.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:33 smithi060 ceph-mon[96524]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-07-20T01:25:33.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:33 smithi183 ceph-mon[98477]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:33.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:33 smithi183 ceph-mon[98477]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-20T01:25:33.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:33 smithi183 ceph-mon[98477]: osdmap e48: 8 total, 7 up, 8 in 2023-07-20T01:25:33.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:33 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:25:33.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:33 smithi183 ceph-mon[98477]: from='osd.7 [v2:172.21.15.136:6816/404531062,v1:172.21.15.136:6817/404531062]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-07-20T01:25:33.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:33 smithi183 ceph-mon[98477]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-07-20T01:25:34.162 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi183:vg_nvme/lv_2 2023-07-20T01:25:34.717 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T01:25:34.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:34 smithi060 ceph-mon[96524]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2023-07-20T01:25:34.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:34 smithi060 ceph-mon[96524]: osdmap e49: 8 total, 7 up, 8 in 2023-07-20T01:25:34.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:34 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:25:34.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:34 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:25:34.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:34 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:34.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:34 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:34.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:34 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:25:34.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:34 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:25:34.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:34 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:25:34.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:34 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:34.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:34 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:34.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:34 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:25:34.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:34 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:34.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:34 smithi183 ceph-mon[98477]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2023-07-20T01:25:34.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:34 smithi183 ceph-mon[98477]: osdmap e49: 8 total, 7 up, 8 in 2023-07-20T01:25:34.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:34 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:25:34.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:34 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:25:34.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:34 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:34.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:34 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:34.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:34 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:25:34.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:34 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:25:34.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:34 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:25:34.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:34 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:34.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:34 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:34.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:34 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:25:34.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:34 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:35.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:34 smithi136 ceph-mon[98083]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2023-07-20T01:25:35.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:34 smithi136 ceph-mon[98083]: osdmap e49: 8 total, 7 up, 8 in 2023-07-20T01:25:35.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:34 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:25:35.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:34 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:25:35.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:34 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:35.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:34 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:35.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:34 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:25:35.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:34 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:25:35.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:34 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:25:35.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:34 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:35.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:34 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:35.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:34 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:25:35.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:34 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:35.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:35 smithi060 ceph-mon[96524]: purged_snaps scrub starts 2023-07-20T01:25:35.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:35 smithi060 ceph-mon[96524]: purged_snaps scrub ok 2023-07-20T01:25:35.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:35 smithi060 ceph-mon[96524]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:35.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:35 smithi060 ceph-mon[96524]: Detected new or changed devices on smithi136 2023-07-20T01:25:35.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:35 smithi060 ceph-mon[96524]: Adjusting osd_memory_target on smithi136 to 5722M 2023-07-20T01:25:35.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:35 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:25:35.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:35 smithi060 ceph-mon[96524]: osd.7 [v2:172.21.15.136:6816/404531062,v1:172.21.15.136:6817/404531062] boot 2023-07-20T01:25:35.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:35 smithi060 ceph-mon[96524]: osdmap e50: 8 total, 8 up, 8 in 2023-07-20T01:25:35.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:35 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:25:35.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:35 smithi183 ceph-mon[98477]: purged_snaps scrub starts 2023-07-20T01:25:35.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:35 smithi183 ceph-mon[98477]: purged_snaps scrub ok 2023-07-20T01:25:35.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:35 smithi183 ceph-mon[98477]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:35.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:35 smithi183 ceph-mon[98477]: Detected new or changed devices on smithi136 2023-07-20T01:25:35.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:35 smithi183 ceph-mon[98477]: Adjusting osd_memory_target on smithi136 to 5722M 2023-07-20T01:25:35.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:35 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:25:35.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:35 smithi183 ceph-mon[98477]: osd.7 [v2:172.21.15.136:6816/404531062,v1:172.21.15.136:6817/404531062] boot 2023-07-20T01:25:35.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:35 smithi183 ceph-mon[98477]: osdmap e50: 8 total, 8 up, 8 in 2023-07-20T01:25:35.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:35 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:25:36.022 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.043+0000 7fb4f2730700 1 -- 172.21.15.183:0/1384756334 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb4ec100310 msgr2=0x7fb4ec107bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:25:36.022 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.043+0000 7fb4f2730700 1 --2- 172.21.15.183:0/1384756334 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb4ec100310 0x7fb4ec107bb0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb4e400b0a0 tx=0x7fb4e400b3b0 comp rx=0 tx=0).stop 2023-07-20T01:25:36.022 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.043+0000 7fb4f2730700 1 -- 172.21.15.183:0/1384756334 shutdown_connections 2023-07-20T01:25:36.023 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.043+0000 7fb4f2730700 1 --2- 172.21.15.183:0/1384756334 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb4ec100310 0x7fb4ec107bb0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:25:36.023 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.043+0000 7fb4f2730700 1 --2- 172.21.15.183:0/1384756334 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb4ec100d80 0x7fb4ec1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:25:36.023 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.043+0000 7fb4f2730700 1 --2- 172.21.15.183:0/1384756334 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4ec0ff950 0x7fb4ec0ffdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:25:36.023 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.043+0000 7fb4f2730700 1 -- 172.21.15.183:0/1384756334 >> 172.21.15.183:0/1384756334 conn(0x7fb4ec0fa420 msgr2=0x7fb4ec0fc860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:25:36.024 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.043+0000 7fb4f2730700 1 -- 172.21.15.183:0/1384756334 shutdown_connections 2023-07-20T01:25:36.024 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.043+0000 7fb4f2730700 1 -- 172.21.15.183:0/1384756334 wait complete. 2023-07-20T01:25:36.027 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.044+0000 7fb4f2730700 1 Processor -- start 2023-07-20T01:25:36.027 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.044+0000 7fb4f2730700 1 -- start start 2023-07-20T01:25:36.027 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.044+0000 7fb4f2730700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb4ec0ff950 0x7fb4ec0ff3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:25:36.027 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.044+0000 7fb4f2730700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb4ec100310 0x7fb4ec19b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:25:36.027 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.044+0000 7fb4f2730700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4ec100d80 0x7fb4ec19d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:25:36.028 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.044+0000 7fb4f2730700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ec19ded0 con 0x7fb4ec100d80 2023-07-20T01:25:36.028 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.044+0000 7fb4f2730700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ec19e040 con 0x7fb4ec100310 2023-07-20T01:25:36.028 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.044+0000 7fb4f2730700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ec19e200 con 0x7fb4ec0ff950 2023-07-20T01:25:36.028 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.044+0000 7fb4ebfff700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb4ec0ff950 0x7fb4ec0ff3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:25:36.028 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.044+0000 7fb4ebfff700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb4ec0ff950 0x7fb4ec0ff3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:46056/0 (socket says 172.21.15.183:46056) 2023-07-20T01:25:36.028 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.044+0000 7fb4ebfff700 1 -- 172.21.15.183:0/719854855 learned_addr learned my addr 172.21.15.183:0/719854855 (peer_addr_for_me v2:172.21.15.183:0/0) 2023-07-20T01:25:36.028 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.044+0000 7fb4eb7fe700 1 --2- 172.21.15.183:0/719854855 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb4ec100310 0x7fb4ec19b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:25:36.029 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.044+0000 7fb4f0ccd700 1 --2- 172.21.15.183:0/719854855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4ec100d80 0x7fb4ec19d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:25:36.029 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.044+0000 7fb4ebfff700 1 -- 172.21.15.183:0/719854855 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb4ec100310 msgr2=0x7fb4ec19b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:25:36.029 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.044+0000 7fb4ebfff700 1 --2- 172.21.15.183:0/719854855 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb4ec100310 0x7fb4ec19b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:25:36.029 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.044+0000 7fb4ebfff700 1 -- 172.21.15.183:0/719854855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4ec100d80 msgr2=0x7fb4ec19d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:25:36.029 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.044+0000 7fb4ebfff700 1 --2- 172.21.15.183:0/719854855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4ec100d80 0x7fb4ec19d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:25:36.029 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.045+0000 7fb4ebfff700 1 -- 172.21.15.183:0/719854855 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e4009d00 con 0x7fb4ec0ff950 2023-07-20T01:25:36.029 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.045+0000 7fb4f0ccd700 1 --2- 172.21.15.183:0/719854855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4ec100d80 0x7fb4ec19d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T01:25:36.030 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.045+0000 7fb4ebfff700 1 --2- 172.21.15.183:0/719854855 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb4ec0ff950 0x7fb4ec0ff3c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc00c9b0 tx=0x7fb4dc00ccc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:25:36.030 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.045+0000 7fb4e97fa700 1 -- 172.21.15.183:0/719854855 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc007b00 con 0x7fb4ec0ff950 2023-07-20T01:25:36.030 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.045+0000 7fb4f2730700 1 -- 172.21.15.183:0/719854855 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4ec19e4e0 con 0x7fb4ec0ff950 2023-07-20T01:25:36.030 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.045+0000 7fb4e97fa700 1 -- 172.21.15.183:0/719854855 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb4dc007c60 con 0x7fb4ec0ff950 2023-07-20T01:25:36.030 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.045+0000 7fb4e97fa700 1 -- 172.21.15.183:0/719854855 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc018660 con 0x7fb4ec0ff950 2023-07-20T01:25:36.030 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.045+0000 7fb4f2730700 1 -- 172.21.15.183:0/719854855 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4ec19ea30 con 0x7fb4ec0ff950 2023-07-20T01:25:36.031 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.046+0000 7fb4f2730700 1 -- 172.21.15.183:0/719854855 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4ec10adc0 con 0x7fb4ec0ff950 2023-07-20T01:25:36.031 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.047+0000 7fb4e97fa700 1 -- 172.21.15.183:0/719854855 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb4dc00f450 con 0x7fb4ec0ff950 2023-07-20T01:25:36.031 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.047+0000 7fb4e97fa700 1 --2- 172.21.15.183:0/719854855 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fb4d40a1150 0x7fb4d40a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:25:36.031 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.047+0000 7fb4e97fa700 1 -- 172.21.15.183:0/719854855 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5426+0+0 (secure 0 0 0) 0x7fb4dc0cc080 con 0x7fb4ec0ff950 2023-07-20T01:25:36.031 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.048+0000 7fb4eb7fe700 1 --2- 172.21.15.183:0/719854855 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fb4d40a1150 0x7fb4d40a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:25:36.031 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.048+0000 7fb4eb7fe700 1 --2- 172.21.15.183:0/719854855 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fb4d40a1150 0x7fb4d40a3610 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec0ff920 tx=0x7fb4e000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:25:36.032 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.051+0000 7fb4e97fa700 1 -- 172.21.15.183:0/719854855 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb4dc0963a0 con 0x7fb4ec0ff950 2023-07-20T01:25:36.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:35 smithi136 ceph-mon[98083]: purged_snaps scrub starts 2023-07-20T01:25:36.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:35 smithi136 ceph-mon[98083]: purged_snaps scrub ok 2023-07-20T01:25:36.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:35 smithi136 ceph-mon[98083]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-20T01:25:36.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:35 smithi136 ceph-mon[98083]: Detected new or changed devices on smithi136 2023-07-20T01:25:36.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:35 smithi136 ceph-mon[98083]: Adjusting osd_memory_target on smithi136 to 5722M 2023-07-20T01:25:36.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:35 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:25:36.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:35 smithi136 ceph-mon[98083]: osd.7 [v2:172.21.15.136:6816/404531062,v1:172.21.15.136:6817/404531062] boot 2023-07-20T01:25:36.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:35 smithi136 ceph-mon[98083]: osdmap e50: 8 total, 8 up, 8 in 2023-07-20T01:25:36.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:35 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-20T01:25:36.221 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:36.242+0000 7fb4f2730700 1 -- 172.21.15.183:0/719854855 --> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fb4ec060bb0 con 0x7fb4d40a1150 2023-07-20T01:25:36.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:36 smithi060 ceph-mon[96524]: osdmap e51: 8 total, 8 up, 8 in 2023-07-20T01:25:36.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:36 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:25:36.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:36 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:25:36.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:36 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:36.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:36 smithi183 ceph-mon[98477]: osdmap e51: 8 total, 8 up, 8 in 2023-07-20T01:25:36.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:36 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:25:36.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:36 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:25:36.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:36 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:37.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:36 smithi136 ceph-mon[98083]: osdmap e51: 8 total, 8 up, 8 in 2023-07-20T01:25:37.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:36 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:25:37.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:36 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:25:37.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:36 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:37.672 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:37 smithi183 ceph-mon[98477]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:37.925 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:25:37 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:25:37.833+0000 7f2cff960700 -1 calc_pg_upmaps abort due to max <= 0 2023-07-20T01:25:37.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:37 smithi060 ceph-mon[96524]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:37.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:37 smithi060 ceph-mon[96524]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:25:37.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:37 smithi060 ceph-mon[96524]: osdmap e52: 8 total, 8 up, 8 in 2023-07-20T01:25:37.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:37 smithi183 ceph-mon[98477]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:25:37.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:37 smithi183 ceph-mon[98477]: osdmap e52: 8 total, 8 up, 8 in 2023-07-20T01:25:38.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:37 smithi136 ceph-mon[98083]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:38.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:37 smithi136 ceph-mon[98083]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:25:38.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:37 smithi136 ceph-mon[98083]: osdmap e52: 8 total, 8 up, 8 in 2023-07-20T01:25:38.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:38 smithi183 ceph-mon[98477]: osdmap e53: 8 total, 8 up, 8 in 2023-07-20T01:25:38.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:38 smithi183 ceph-mon[98477]: from='client.? 172.21.15.183:0/393447196' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "68b0051c-0fd3-4ea1-b7df-ea0b8409b8cc"}]: dispatch 2023-07-20T01:25:38.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:38 smithi183 ceph-mon[98477]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "68b0051c-0fd3-4ea1-b7df-ea0b8409b8cc"}]: dispatch 2023-07-20T01:25:38.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:38 smithi183 ceph-mon[98477]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "68b0051c-0fd3-4ea1-b7df-ea0b8409b8cc"}]': finished 2023-07-20T01:25:38.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:38 smithi183 ceph-mon[98477]: osdmap e54: 9 total, 8 up, 9 in 2023-07-20T01:25:38.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:38 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:25:39.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:38 smithi136 ceph-mon[98083]: osdmap e53: 8 total, 8 up, 8 in 2023-07-20T01:25:39.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:38 smithi136 ceph-mon[98083]: from='client.? 172.21.15.183:0/393447196' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "68b0051c-0fd3-4ea1-b7df-ea0b8409b8cc"}]: dispatch 2023-07-20T01:25:39.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:38 smithi136 ceph-mon[98083]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "68b0051c-0fd3-4ea1-b7df-ea0b8409b8cc"}]: dispatch 2023-07-20T01:25:39.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:38 smithi136 ceph-mon[98083]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "68b0051c-0fd3-4ea1-b7df-ea0b8409b8cc"}]': finished 2023-07-20T01:25:39.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:38 smithi136 ceph-mon[98083]: osdmap e54: 9 total, 8 up, 9 in 2023-07-20T01:25:39.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:38 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:25:39.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:38 smithi060 ceph-mon[96524]: osdmap e53: 8 total, 8 up, 8 in 2023-07-20T01:25:39.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:38 smithi060 ceph-mon[96524]: from='client.? 172.21.15.183:0/393447196' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "68b0051c-0fd3-4ea1-b7df-ea0b8409b8cc"}]: dispatch 2023-07-20T01:25:39.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:38 smithi060 ceph-mon[96524]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "68b0051c-0fd3-4ea1-b7df-ea0b8409b8cc"}]: dispatch 2023-07-20T01:25:39.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:38 smithi060 ceph-mon[96524]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "68b0051c-0fd3-4ea1-b7df-ea0b8409b8cc"}]': finished 2023-07-20T01:25:39.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:38 smithi060 ceph-mon[96524]: osdmap e54: 9 total, 8 up, 9 in 2023-07-20T01:25:39.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:38 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:25:39.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:39 smithi183 ceph-mon[98477]: pgmap v163: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:39.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:39 smithi183 ceph-mon[98477]: from='client.? 172.21.15.183:0/4177071096' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:25:40.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:39 smithi136 ceph-mon[98083]: pgmap v163: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:40.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:39 smithi136 ceph-mon[98083]: from='client.? 172.21.15.183:0/4177071096' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:25:40.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:39 smithi060 ceph-mon[96524]: pgmap v163: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:40.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:39 smithi060 ceph-mon[96524]: from='client.? 172.21.15.183:0/4177071096' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:25:41.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:41 smithi183 ceph-mon[98477]: pgmap v165: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:42.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:41 smithi136 ceph-mon[98083]: pgmap v165: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:42.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:41 smithi060 ceph-mon[96524]: pgmap v165: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:43.756 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:43 smithi183 ceph-mon[98477]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:44.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:43 smithi136 ceph-mon[98083]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:44.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:43 smithi060 ceph-mon[96524]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:45.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:45 smithi183 ceph-mon[98477]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:45.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:45 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-20T01:25:45.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:45 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:46.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:45 smithi136 ceph-mon[98083]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:46.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:45 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-20T01:25:46.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:45 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:46.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:45 smithi060 ceph-mon[96524]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:46.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:45 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-20T01:25:46.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:45 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:46.916 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:46 smithi183 ceph-mon[98477]: Deploying daemon osd.8 on smithi183 2023-07-20T01:25:47.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:46 smithi136 ceph-mon[98083]: Deploying daemon osd.8 on smithi183 2023-07-20T01:25:47.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:46 smithi060 ceph-mon[96524]: Deploying daemon osd.8 on smithi183 2023-07-20T01:25:47.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:47 smithi183 ceph-mon[98477]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:48.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:47 smithi136 ceph-mon[98083]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:48.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:47 smithi060 ceph-mon[96524]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:50.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:49 smithi136 ceph-mon[98083]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:50.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:49 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:50.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:49 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:50.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:49 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:25:50.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:49 smithi060 ceph-mon[96524]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:50.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:49 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:50.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:49 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:50.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:49 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:25:50.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:49 smithi183 ceph-mon[98477]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:50.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:49 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:50.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:49 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:50.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:49 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:25:52.038 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:51 smithi183 ceph-mon[98477]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:52.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:51 smithi136 ceph-mon[98083]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:52.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:51 smithi060 ceph-mon[96524]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:52.337 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:52.358+0000 7fb4e97fa700 1 -- 172.21.15.183:0/719854855 <== mgr.14152 v2:172.21.15.60:6800/1961833180 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb4ec060bb0 con 0x7fb4d40a1150 2023-07-20T01:25:52.338 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 8 on host 'smithi183' 2023-07-20T01:25:52.338 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:52.360+0000 7fb4f2730700 1 -- 172.21.15.183:0/719854855 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fb4d40a1150 msgr2=0x7fb4d40a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:25:52.339 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:52.360+0000 7fb4f2730700 1 --2- 172.21.15.183:0/719854855 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fb4d40a1150 0x7fb4d40a3610 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec0ff920 tx=0x7fb4e000a380 comp rx=0 tx=0).stop 2023-07-20T01:25:52.339 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:52.360+0000 7fb4f2730700 1 -- 172.21.15.183:0/719854855 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb4ec0ff950 msgr2=0x7fb4ec0ff3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:25:52.339 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:52.360+0000 7fb4f2730700 1 --2- 172.21.15.183:0/719854855 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb4ec0ff950 0x7fb4ec0ff3c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc00c9b0 tx=0x7fb4dc00ccc0 comp rx=0 tx=0).stop 2023-07-20T01:25:52.339 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:52.360+0000 7fb4f2730700 1 -- 172.21.15.183:0/719854855 shutdown_connections 2023-07-20T01:25:52.339 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:52.360+0000 7fb4f2730700 1 --2- 172.21.15.183:0/719854855 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fb4d40a1150 0x7fb4d40a3610 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:25:52.339 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:52.360+0000 7fb4f2730700 1 --2- 172.21.15.183:0/719854855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4ec100d80 0x7fb4ec19d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:25:52.340 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:52.360+0000 7fb4f2730700 1 --2- 172.21.15.183:0/719854855 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb4ec0ff950 0x7fb4ec0ff3c0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:25:52.340 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:52.360+0000 7fb4f2730700 1 --2- 172.21.15.183:0/719854855 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb4ec100310 0x7fb4ec19b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:25:52.340 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:52.360+0000 7fb4f2730700 1 -- 172.21.15.183:0/719854855 >> 172.21.15.183:0/719854855 conn(0x7fb4ec0fa420 msgr2=0x7fb4ec0fc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:25:52.340 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:52.360+0000 7fb4f2730700 1 -- 172.21.15.183:0/719854855 shutdown_connections 2023-07-20T01:25:52.340 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:25:52.360+0000 7fb4f2730700 1 -- 172.21.15.183:0/719854855 wait complete. 2023-07-20T01:25:52.696 DEBUG:teuthology.orchestra.run.smithi183:osd.8> sudo journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.8.service 2023-07-20T01:25:52.698 INFO:tasks.cephadm:Deploying osd.9 on smithi060 with /dev/vg_nvme/lv_1... 2023-07-20T01:25:52.699 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-20T01:25:52.733 INFO:journalctl@ceph.osd.8.smithi183.stdout:-- Logs begin at Thu 2023-07-20 01:00:08 UTC. -- 2023-07-20T01:25:53.247 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:53 smithi183 ceph-mon[98477]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:53.247 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:53 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:53.247 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:53 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:53.247 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:53 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:53.247 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:53 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:25:53.247 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:53 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:53.248 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:53 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:53.248 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:53 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:25:53.248 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:53 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:53.277 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:25:53.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:53 smithi136 ceph-mon[98083]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:53.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:53 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:53.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:53 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:53.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:53 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:53.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:53 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:25:53.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:53 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:53.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:53 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:53.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:53 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:25:53.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:53 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:53.425 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:53 smithi060 ceph-mon[96524]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:53.425 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:53 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:53.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:53 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:53.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:53 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:53.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:53 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:25:53.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:53 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:53.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:53 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:53.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:53 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:25:53.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:53 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:53.562 INFO:journalctl@ceph.osd.8.smithi183.stdout:Jul 20 01:25:53 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-8[115251]: 2023-07-20T01:25:53.500+0000 7fd550a16640 -1 osd.8 0 log_to_monitors true 2023-07-20T01:25:54.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:54 smithi060 ceph-mon[96524]: from='osd.8 [v2:172.21.15.183:6816/502824503,v1:172.21.15.183:6817/502824503]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T01:25:54.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:54 smithi136 ceph-mon[98083]: from='osd.8 [v2:172.21.15.183:6816/502824503,v1:172.21.15.183:6817/502824503]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T01:25:54.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:54 smithi183 ceph-mon[98477]: from='osd.8 [v2:172.21.15.183:6816/502824503,v1:172.21.15.183:6817/502824503]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-20T01:25:54.796 INFO:teuthology.orchestra.run.smithi060.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-20T01:25:54.797 INFO:teuthology.orchestra.run.smithi060.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-20T01:25:54.797 INFO:teuthology.orchestra.run.smithi060.stderr: stderr: 10+0 records in 2023-07-20T01:25:54.797 INFO:teuthology.orchestra.run.smithi060.stderr:10+0 records out 2023-07-20T01:25:54.797 INFO:teuthology.orchestra.run.smithi060.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0138008 s, 760 MB/s 2023-07-20T01:25:54.797 INFO:teuthology.orchestra.run.smithi060.stderr:--> Zapping successful for: 2023-07-20T01:25:55.167 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:55 smithi060 ceph-mon[96524]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:55.168 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:55 smithi060 ceph-mon[96524]: from='osd.8 [v2:172.21.15.183:6816/502824503,v1:172.21.15.183:6817/502824503]' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-20T01:25:55.168 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:55 smithi060 ceph-mon[96524]: osdmap e55: 9 total, 8 up, 9 in 2023-07-20T01:25:55.168 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:55 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:25:55.168 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:55 smithi060 ceph-mon[96524]: from='osd.8 [v2:172.21.15.183:6816/502824503,v1:172.21.15.183:6817/502824503]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2023-07-20T01:25:55.169 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:55 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:55.169 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:55 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:55.169 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:55 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:25:55.170 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:55 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:25:55.170 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:55 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:25:55.170 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:55 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:55.170 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:55 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:55.171 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:55 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:25:55.171 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:55 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:55.198 INFO:journalctl@ceph.osd.8.smithi183.stdout:Jul 20 01:25:55 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-8[115251]: 2023-07-20T01:25:55.027+0000 7fd54608e700 -1 osd.8 0 waiting for initial osdmap 2023-07-20T01:25:55.198 INFO:journalctl@ceph.osd.8.smithi183.stdout:Jul 20 01:25:55 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-8[115251]: 2023-07-20T01:25:55.038+0000 7fd53fe80700 -1 osd.8 56 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T01:25:55.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:55 smithi183 ceph-mon[98477]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:55.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:55 smithi183 ceph-mon[98477]: from='osd.8 [v2:172.21.15.183:6816/502824503,v1:172.21.15.183:6817/502824503]' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-20T01:25:55.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:55 smithi183 ceph-mon[98477]: osdmap e55: 9 total, 8 up, 9 in 2023-07-20T01:25:55.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:55 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:25:55.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:55 smithi183 ceph-mon[98477]: from='osd.8 [v2:172.21.15.183:6816/502824503,v1:172.21.15.183:6817/502824503]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2023-07-20T01:25:55.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:55 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:55.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:55 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:55.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:55 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:25:55.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:55 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:25:55.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:55 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:25:55.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:55 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:55.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:55 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:55.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:55 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:25:55.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:55 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:55.238 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi060:vg_nvme/lv_1 2023-07-20T01:25:55.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:55 smithi136 ceph-mon[98083]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:55.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:55 smithi136 ceph-mon[98083]: from='osd.8 [v2:172.21.15.183:6816/502824503,v1:172.21.15.183:6817/502824503]' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-20T01:25:55.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:55 smithi136 ceph-mon[98083]: osdmap e55: 9 total, 8 up, 9 in 2023-07-20T01:25:55.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:55 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:25:55.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:55 smithi136 ceph-mon[98083]: from='osd.8 [v2:172.21.15.183:6816/502824503,v1:172.21.15.183:6817/502824503]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2023-07-20T01:25:55.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:55 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:55.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:55 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:55.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:55 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:25:55.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:55 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:25:55.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:55 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:25:55.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:55 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:55.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:55 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:55.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:55 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:25:55.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:55 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:25:55.798 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:25:56.299 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:56 smithi060 ceph-mon[96524]: Detected new or changed devices on smithi183 2023-07-20T01:25:56.299 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:56 smithi060 ceph-mon[96524]: Adjusting osd_memory_target on smithi183 to 5722M 2023-07-20T01:25:56.299 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:56 smithi060 ceph-mon[96524]: from='osd.8 [v2:172.21.15.183:6816/502824503,v1:172.21.15.183:6817/502824503]' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2023-07-20T01:25:56.299 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:56 smithi060 ceph-mon[96524]: osdmap e56: 9 total, 8 up, 9 in 2023-07-20T01:25:56.299 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:56 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:25:56.300 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:56 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:25:56.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:56 smithi136 ceph-mon[98083]: Detected new or changed devices on smithi183 2023-07-20T01:25:56.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:56 smithi136 ceph-mon[98083]: Adjusting osd_memory_target on smithi183 to 5722M 2023-07-20T01:25:56.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:56 smithi136 ceph-mon[98083]: from='osd.8 [v2:172.21.15.183:6816/502824503,v1:172.21.15.183:6817/502824503]' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2023-07-20T01:25:56.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:56 smithi136 ceph-mon[98083]: osdmap e56: 9 total, 8 up, 9 in 2023-07-20T01:25:56.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:56 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:25:56.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:56 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:25:56.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:56 smithi183 ceph-mon[98477]: Detected new or changed devices on smithi183 2023-07-20T01:25:56.449 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:56 smithi183 ceph-mon[98477]: Adjusting osd_memory_target on smithi183 to 5722M 2023-07-20T01:25:56.449 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:56 smithi183 ceph-mon[98477]: from='osd.8 [v2:172.21.15.183:6816/502824503,v1:172.21.15.183:6817/502824503]' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2023-07-20T01:25:56.449 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:56 smithi183 ceph-mon[98477]: osdmap e56: 9 total, 8 up, 9 in 2023-07-20T01:25:56.449 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:56 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:25:56.449 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:56 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:25:57.008 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.028+0000 7f30e691f700 1 -- 172.21.15.60:0/3973512297 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30e0101a00 msgr2=0x7f30e0101e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:25:57.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.028+0000 7f30e691f700 1 --2- 172.21.15.60:0/3973512297 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30e0101a00 0x7f30e0101e60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f30d40099c0 tx=0x7f30d4009cd0 comp rx=0 tx=0).stop 2023-07-20T01:25:57.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.028+0000 7f30e691f700 1 -- 172.21.15.60:0/3973512297 shutdown_connections 2023-07-20T01:25:57.009 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.028+0000 7f30e691f700 1 --2- 172.21.15.60:0/3973512297 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f30e01023a0 0x7f30e0108c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:25:57.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.028+0000 7f30e691f700 1 --2- 172.21.15.60:0/3973512297 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f30e0100800 0x7f30e0100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:25:57.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.028+0000 7f30e691f700 1 --2- 172.21.15.60:0/3973512297 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30e0101a00 0x7f30e0101e60 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:25:57.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.028+0000 7f30e691f700 1 -- 172.21.15.60:0/3973512297 >> 172.21.15.60:0/3973512297 conn(0x7f30e00fbd60 msgr2=0x7f30e00fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:25:57.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.029+0000 7f30e691f700 1 -- 172.21.15.60:0/3973512297 shutdown_connections 2023-07-20T01:25:57.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.029+0000 7f30e691f700 1 -- 172.21.15.60:0/3973512297 wait complete. 2023-07-20T01:25:57.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.029+0000 7f30e691f700 1 Processor -- start 2023-07-20T01:25:57.010 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.029+0000 7f30e691f700 1 -- start start 2023-07-20T01:25:57.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.030+0000 7f30e691f700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f30e0100800 0x7f30e0074510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:25:57.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.030+0000 7f30e691f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30e0101a00 0x7f30e0072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:25:57.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.030+0000 7f30e691f700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f30e01023a0 0x7f30e00730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:25:57.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.030+0000 7f30e691f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e0073820 con 0x7f30e0101a00 2023-07-20T01:25:57.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.030+0000 7f30e691f700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e0073990 con 0x7f30e01023a0 2023-07-20T01:25:57.011 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.030+0000 7f30e691f700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e01a0410 con 0x7f30e0100800 2023-07-20T01:25:57.012 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.030+0000 7f30df7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30e0101a00 0x7f30e0072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:25:57.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.030+0000 7f30dffff700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f30e0100800 0x7f30e0074510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:25:57.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.030+0000 7f30df7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30e0101a00 0x7f30e0072b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42576/0 (socket says 172.21.15.60:42576) 2023-07-20T01:25:57.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.030+0000 7f30df7fe700 1 -- 172.21.15.60:0/2906292057 learned_addr learned my addr 172.21.15.60:0/2906292057 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:25:57.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.030+0000 7f30e4ebc700 1 --2- 172.21.15.60:0/2906292057 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f30e01023a0 0x7f30e00730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:25:57.013 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.030+0000 7f30df7fe700 1 -- 172.21.15.60:0/2906292057 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f30e0100800 msgr2=0x7f30e0074510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:25:57.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.030+0000 7f30df7fe700 1 --2- 172.21.15.60:0/2906292057 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f30e0100800 0x7f30e0074510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:25:57.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.030+0000 7f30df7fe700 1 -- 172.21.15.60:0/2906292057 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f30e01023a0 msgr2=0x7f30e00730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:25:57.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.030+0000 7f30df7fe700 1 --2- 172.21.15.60:0/2906292057 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f30e01023a0 0x7f30e00730a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:25:57.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.030+0000 7f30df7fe700 1 -- 172.21.15.60:0/2906292057 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d40095f0 con 0x7f30e0101a00 2023-07-20T01:25:57.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.030+0000 7f30df7fe700 1 --2- 172.21.15.60:0/2906292057 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30e0101a00 0x7f30e0072b60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f30d4009fd0 tx=0x7f30d400fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:25:57.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.033+0000 7f30dd7fa700 1 -- 172.21.15.60:0/2906292057 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d401d070 con 0x7f30e0101a00 2023-07-20T01:25:57.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.033+0000 7f30e691f700 1 -- 172.21.15.60:0/2906292057 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30e01a0690 con 0x7f30e0101a00 2023-07-20T01:25:57.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.033+0000 7f30e691f700 1 -- 172.21.15.60:0/2906292057 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30e01a0b80 con 0x7f30e0101a00 2023-07-20T01:25:57.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.034+0000 7f30dd7fa700 1 -- 172.21.15.60:0/2906292057 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f30d400fe70 con 0x7f30e0101a00 2023-07-20T01:25:57.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.034+0000 7f30dd7fa700 1 -- 172.21.15.60:0/2906292057 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d402b610 con 0x7f30e0101a00 2023-07-20T01:25:57.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.034+0000 7f30e691f700 1 -- 172.21.15.60:0/2906292057 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30cc005320 con 0x7f30e0101a00 2023-07-20T01:25:57.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.036+0000 7f30dd7fa700 1 -- 172.21.15.60:0/2906292057 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f30d4003760 con 0x7f30e0101a00 2023-07-20T01:25:57.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.037+0000 7f30dd7fa700 1 --2- 172.21.15.60:0/2906292057 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f30c80a1150 0x7f30c80a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:25:57.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.037+0000 7f30dffff700 1 --2- 172.21.15.60:0/2906292057 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f30c80a1150 0x7f30c80a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:25:57.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.037+0000 7f30dd7fa700 1 -- 172.21.15.60:0/2906292057 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5858+0+0 (secure 0 0 0) 0x7f30d40ccef0 con 0x7f30e0101a00 2023-07-20T01:25:57.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.037+0000 7f30dffff700 1 --2- 172.21.15.60:0/2906292057 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f30c80a1150 0x7f30c80a3610 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f30e0101860 tx=0x7f30d0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:25:57.019 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.040+0000 7f30dd7fa700 1 -- 172.21.15.60:0/2906292057 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f30d4096fc0 con 0x7f30e0101a00 2023-07-20T01:25:57.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:57 smithi060 ceph-mon[96524]: purged_snaps scrub starts 2023-07-20T01:25:57.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:57 smithi060 ceph-mon[96524]: purged_snaps scrub ok 2023-07-20T01:25:57.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:57 smithi060 ceph-mon[96524]: pgmap v175: 1 pgs: 1 peering; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:57.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:57 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:25:57.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:57 smithi060 ceph-mon[96524]: osd.8 [v2:172.21.15.183:6816/502824503,v1:172.21.15.183:6817/502824503] boot 2023-07-20T01:25:57.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:57 smithi060 ceph-mon[96524]: osdmap e57: 9 total, 9 up, 9 in 2023-07-20T01:25:57.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:57 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:25:57.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:57 smithi060 ceph-mon[96524]: osdmap e58: 9 total, 9 up, 9 in 2023-07-20T01:25:57.224 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:25:57.245+0000 7f30e691f700 1 -- 172.21.15.60:0/2906292057 --> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f30cc000bf0 con 0x7f30c80a1150 2023-07-20T01:25:57.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:57 smithi136 ceph-mon[98083]: purged_snaps scrub starts 2023-07-20T01:25:57.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:57 smithi136 ceph-mon[98083]: purged_snaps scrub ok 2023-07-20T01:25:57.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:57 smithi136 ceph-mon[98083]: pgmap v175: 1 pgs: 1 peering; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:57.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:57 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:25:57.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:57 smithi136 ceph-mon[98083]: osd.8 [v2:172.21.15.183:6816/502824503,v1:172.21.15.183:6817/502824503] boot 2023-07-20T01:25:57.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:57 smithi136 ceph-mon[98083]: osdmap e57: 9 total, 9 up, 9 in 2023-07-20T01:25:57.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:57 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:25:57.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:57 smithi136 ceph-mon[98083]: osdmap e58: 9 total, 9 up, 9 in 2023-07-20T01:25:57.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:57 smithi183 ceph-mon[98477]: purged_snaps scrub starts 2023-07-20T01:25:57.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:57 smithi183 ceph-mon[98477]: purged_snaps scrub ok 2023-07-20T01:25:57.449 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:57 smithi183 ceph-mon[98477]: pgmap v175: 1 pgs: 1 peering; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-20T01:25:57.449 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:57 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:25:57.449 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:57 smithi183 ceph-mon[98477]: osd.8 [v2:172.21.15.183:6816/502824503,v1:172.21.15.183:6817/502824503] boot 2023-07-20T01:25:57.449 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:57 smithi183 ceph-mon[98477]: osdmap e57: 9 total, 9 up, 9 in 2023-07-20T01:25:57.449 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:57 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-20T01:25:57.449 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:57 smithi183 ceph-mon[98477]: osdmap e58: 9 total, 9 up, 9 in 2023-07-20T01:25:58.340 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:58 smithi060 ceph-mon[96524]: from='client.14415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:25:58.340 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:58 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:25:58.340 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:58 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:25:58.340 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:58 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:58.341 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:58 smithi060 ceph-mon[96524]: osdmap e59: 9 total, 9 up, 9 in 2023-07-20T01:25:58.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:58 smithi136 ceph-mon[98083]: from='client.14415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:25:58.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:58 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:25:58.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:58 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:25:58.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:58 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:58.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:58 smithi136 ceph-mon[98083]: osdmap e59: 9 total, 9 up, 9 in 2023-07-20T01:25:58.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:58 smithi183 ceph-mon[98477]: from='client.14415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:25:58.449 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:58 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:25:58.449 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:58 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:25:58.449 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:58 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:25:58.449 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:58 smithi183 ceph-mon[98477]: osdmap e59: 9 total, 9 up, 9 in 2023-07-20T01:25:59.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:59 smithi136 ceph-mon[98083]: pgmap v179: 1 pgs: 1 peering; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2023-07-20T01:25:59.646 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:25:59 smithi136 ceph-mon[98083]: osdmap e60: 9 total, 9 up, 9 in 2023-07-20T01:25:59.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:59 smithi060 ceph-mon[96524]: pgmap v179: 1 pgs: 1 peering; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2023-07-20T01:25:59.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:25:59 smithi060 ceph-mon[96524]: osdmap e60: 9 total, 9 up, 9 in 2023-07-20T01:25:59.698 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:59 smithi183 ceph-mon[98477]: pgmap v179: 1 pgs: 1 peering; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2023-07-20T01:25:59.698 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:25:59 smithi183 ceph-mon[98477]: osdmap e60: 9 total, 9 up, 9 in 2023-07-20T01:26:00.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:00 smithi136 ceph-mon[98083]: osdmap e61: 9 total, 9 up, 9 in 2023-07-20T01:26:00.646 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:00 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/3558337133' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cafd4cc3-258c-4f88-8d3d-0eb4dfa2e2c1"}]: dispatch 2023-07-20T01:26:00.646 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:00 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/3558337133' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cafd4cc3-258c-4f88-8d3d-0eb4dfa2e2c1"}]': finished 2023-07-20T01:26:00.646 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:00 smithi136 ceph-mon[98083]: osdmap e62: 10 total, 9 up, 10 in 2023-07-20T01:26:00.646 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:00 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T01:26:00.646 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:00 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/2018333444' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:26:00.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:00 smithi060 ceph-mon[96524]: osdmap e61: 9 total, 9 up, 9 in 2023-07-20T01:26:00.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:00 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/3558337133' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cafd4cc3-258c-4f88-8d3d-0eb4dfa2e2c1"}]: dispatch 2023-07-20T01:26:00.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:00 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/3558337133' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cafd4cc3-258c-4f88-8d3d-0eb4dfa2e2c1"}]': finished 2023-07-20T01:26:00.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:00 smithi060 ceph-mon[96524]: osdmap e62: 10 total, 9 up, 10 in 2023-07-20T01:26:00.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:00 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T01:26:00.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:00 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/2018333444' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:26:00.698 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:00 smithi183 ceph-mon[98477]: osdmap e61: 9 total, 9 up, 9 in 2023-07-20T01:26:00.698 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:00 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/3558337133' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cafd4cc3-258c-4f88-8d3d-0eb4dfa2e2c1"}]: dispatch 2023-07-20T01:26:00.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:00 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/3558337133' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cafd4cc3-258c-4f88-8d3d-0eb4dfa2e2c1"}]': finished 2023-07-20T01:26:00.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:00 smithi183 ceph-mon[98477]: osdmap e62: 10 total, 9 up, 10 in 2023-07-20T01:26:00.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:00 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T01:26:00.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:00 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/2018333444' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:26:01.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:01 smithi136 ceph-mon[98083]: pgmap v183: 1 pgs: 1 peering; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2023-07-20T01:26:01.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:01 smithi060 ceph-mon[96524]: pgmap v183: 1 pgs: 1 peering; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2023-07-20T01:26:01.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:01 smithi183 ceph-mon[98477]: pgmap v183: 1 pgs: 1 peering; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2023-07-20T01:26:03.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:03 smithi136 ceph-mon[98083]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 129 KiB/s, 0 objects/s recovering 2023-07-20T01:26:03.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:03 smithi060 ceph-mon[96524]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 129 KiB/s, 0 objects/s recovering 2023-07-20T01:26:03.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:03 smithi183 ceph-mon[98477]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 129 KiB/s, 0 objects/s recovering 2023-07-20T01:26:05.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:05 smithi136 ceph-mon[98083]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T01:26:05.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:05 smithi060 ceph-mon[96524]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T01:26:05.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:05 smithi183 ceph-mon[98477]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-20T01:26:06.892 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:06 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-20T01:26:06.893 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:06 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:26:06.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:06 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-20T01:26:06.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:06 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:26:06.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:06 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-20T01:26:06.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:06 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:26:07.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:07 smithi136 ceph-mon[98083]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-20T01:26:07.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:07 smithi136 ceph-mon[98083]: Deploying daemon osd.9 on smithi060 2023-07-20T01:26:07.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:07 smithi060 ceph-mon[96524]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-20T01:26:07.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:07 smithi060 ceph-mon[96524]: Deploying daemon osd.9 on smithi060 2023-07-20T01:26:07.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:07 smithi183 ceph-mon[98477]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-20T01:26:07.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:07 smithi183 ceph-mon[98477]: Deploying daemon osd.9 on smithi060 2023-07-20T01:26:09.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:09 smithi136 ceph-mon[98083]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2023-07-20T01:26:09.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:09 smithi060 ceph-mon[96524]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2023-07-20T01:26:09.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:09 smithi183 ceph-mon[98477]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2023-07-20T01:26:11.896 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:11 smithi060 ceph-mon[96524]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 56 KiB/s, 0 objects/s recovering 2023-07-20T01:26:11.897 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:11 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:11.897 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:11 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:11.897 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:11 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:26:12.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:11 smithi136 ceph-mon[98083]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 56 KiB/s, 0 objects/s recovering 2023-07-20T01:26:12.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:11 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:12.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:11 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:12.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:11 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:26:12.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:11 smithi183 ceph-mon[98477]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 56 KiB/s, 0 objects/s recovering 2023-07-20T01:26:12.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:11 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:12.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:11 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:12.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:11 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:26:13.580 INFO:teuthology.orchestra.run.smithi060.stdout:Created osd(s) 9 on host 'smithi060' 2023-07-20T01:26:13.580 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:13.601+0000 7f30dd7fa700 1 -- 172.21.15.60:0/2906292057 <== mgr.14152 v2:172.21.15.60:6800/1961833180 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f30cc000bf0 con 0x7f30c80a1150 2023-07-20T01:26:13.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:13.604+0000 7f30e691f700 1 -- 172.21.15.60:0/2906292057 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f30c80a1150 msgr2=0x7f30c80a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:26:13.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:13.604+0000 7f30e691f700 1 --2- 172.21.15.60:0/2906292057 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f30c80a1150 0x7f30c80a3610 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f30e0101860 tx=0x7f30d0009260 comp rx=0 tx=0).stop 2023-07-20T01:26:13.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:13.604+0000 7f30e691f700 1 -- 172.21.15.60:0/2906292057 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30e0101a00 msgr2=0x7f30e0072b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:26:13.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:13.604+0000 7f30e691f700 1 --2- 172.21.15.60:0/2906292057 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30e0101a00 0x7f30e0072b60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f30d4009fd0 tx=0x7f30d400fa00 comp rx=0 tx=0).stop 2023-07-20T01:26:13.583 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:13.604+0000 7f30e691f700 1 -- 172.21.15.60:0/2906292057 shutdown_connections 2023-07-20T01:26:13.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:13.604+0000 7f30e691f700 1 --2- 172.21.15.60:0/2906292057 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f30c80a1150 0x7f30c80a3610 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:26:13.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:13.604+0000 7f30e691f700 1 --2- 172.21.15.60:0/2906292057 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f30e01023a0 0x7f30e00730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:26:13.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:13.604+0000 7f30e691f700 1 --2- 172.21.15.60:0/2906292057 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f30e0100800 0x7f30e0074510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:26:13.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:13.604+0000 7f30e691f700 1 --2- 172.21.15.60:0/2906292057 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30e0101a00 0x7f30e0072b60 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:26:13.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:13.604+0000 7f30e691f700 1 -- 172.21.15.60:0/2906292057 >> 172.21.15.60:0/2906292057 conn(0x7f30e00fbd60 msgr2=0x7f30e00fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:26:13.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:13.604+0000 7f30e691f700 1 -- 172.21.15.60:0/2906292057 shutdown_connections 2023-07-20T01:26:13.584 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:13.604+0000 7f30e691f700 1 -- 172.21.15.60:0/2906292057 wait complete. 2023-07-20T01:26:13.990 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:13 smithi060 ceph-mon[96524]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-20T01:26:13.991 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:13 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:13.991 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:13 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:13.991 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:13 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:13.991 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:13 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:13.993 DEBUG:teuthology.orchestra.run.smithi060:osd.9> sudo journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.9.service 2023-07-20T01:26:13.995 INFO:tasks.cephadm:Deploying osd.10 on smithi136 with /dev/vg_nvme/lv_1... 2023-07-20T01:26:13.995 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-20T01:26:14.024 INFO:journalctl@ceph.osd.9.smithi060.stdout:-- Logs begin at Thu 2023-07-20 01:00:02 UTC. -- 2023-07-20T01:26:14.028 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:13 smithi136 ceph-mon[98083]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-20T01:26:14.028 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:13 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:14.028 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:13 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:14.029 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:13 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:14.029 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:13 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:14.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:13 smithi183 ceph-mon[98477]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-20T01:26:14.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:13 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:14.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:13 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:14.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:13 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:14.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:13 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:14.562 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T01:26:14.871 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:14 smithi060 ceph-mon[96524]: from='osd.9 [v2:172.21.15.60:6826/634188302,v1:172.21.15.60:6827/634188302]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-20T01:26:14.872 INFO:journalctl@ceph.osd.9.smithi060.stdout:Jul 20 01:26:14 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-9[125860]: 2023-07-20T01:26:14.653+0000 7f183b0cb640 -1 osd.9 0 log_to_monitors true 2023-07-20T01:26:14.985 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:14 smithi136 ceph-mon[98083]: from='osd.9 [v2:172.21.15.60:6826/634188302,v1:172.21.15.60:6827/634188302]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-20T01:26:15.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:14 smithi183 ceph-mon[98477]: from='osd.9 [v2:172.21.15.60:6826/634188302,v1:172.21.15.60:6827/634188302]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-20T01:26:15.996 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:15 smithi060 ceph-mon[96524]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T01:26:15.996 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:15 smithi060 ceph-mon[96524]: from='osd.9 [v2:172.21.15.60:6826/634188302,v1:172.21.15.60:6827/634188302]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-20T01:26:15.997 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:15 smithi060 ceph-mon[96524]: osdmap e63: 10 total, 9 up, 10 in 2023-07-20T01:26:15.997 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:15 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T01:26:15.997 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:15 smithi060 ceph-mon[96524]: from='osd.9 [v2:172.21.15.60:6826/634188302,v1:172.21.15.60:6827/634188302]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2023-07-20T01:26:15.997 INFO:journalctl@ceph.osd.9.smithi060.stdout:Jul 20 01:26:15 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-9[125860]: 2023-07-20T01:26:15.763+0000 7f1830743700 -1 osd.9 0 waiting for initial osdmap 2023-07-20T01:26:15.997 INFO:journalctl@ceph.osd.9.smithi060.stdout:Jul 20 01:26:15 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-9[125860]: 2023-07-20T01:26:15.775+0000 7f182c539700 -1 osd.9 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T01:26:16.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:15 smithi136 ceph-mon[98083]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T01:26:16.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:15 smithi136 ceph-mon[98083]: from='osd.9 [v2:172.21.15.60:6826/634188302,v1:172.21.15.60:6827/634188302]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-20T01:26:16.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:15 smithi136 ceph-mon[98083]: osdmap e63: 10 total, 9 up, 10 in 2023-07-20T01:26:16.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:15 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T01:26:16.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:15 smithi136 ceph-mon[98083]: from='osd.9 [v2:172.21.15.60:6826/634188302,v1:172.21.15.60:6827/634188302]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2023-07-20T01:26:16.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:15 smithi183 ceph-mon[98477]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T01:26:16.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:15 smithi183 ceph-mon[98477]: from='osd.9 [v2:172.21.15.60:6826/634188302,v1:172.21.15.60:6827/634188302]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-20T01:26:16.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:15 smithi183 ceph-mon[98477]: osdmap e63: 10 total, 9 up, 10 in 2023-07-20T01:26:16.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:15 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T01:26:16.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:15 smithi183 ceph-mon[98477]: from='osd.9 [v2:172.21.15.60:6826/634188302,v1:172.21.15.60:6827/634188302]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2023-07-20T01:26:16.201 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-20T01:26:16.201 INFO:teuthology.orchestra.run.smithi136.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-20T01:26:16.201 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10+0 records in 2023-07-20T01:26:16.201 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2023-07-20T01:26:16.201 INFO:teuthology.orchestra.run.smithi136.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0123388 s, 850 MB/s 2023-07-20T01:26:16.202 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping successful for: 2023-07-20T01:26:16.704 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi136:vg_nvme/lv_1 2023-07-20T01:26:17.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:16 smithi136 ceph-mon[98083]: from='osd.9 [v2:172.21.15.60:6826/634188302,v1:172.21.15.60:6827/634188302]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2023-07-20T01:26:17.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:16 smithi136 ceph-mon[98083]: osdmap e64: 10 total, 9 up, 10 in 2023-07-20T01:26:17.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:16 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T01:26:17.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:16 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T01:26:17.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:16 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:17.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:16 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:17.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:16 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:26:17.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:16 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:26:17.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:16 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:26:17.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:16 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:26:17.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:16 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:17.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:16 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:26:17.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:16 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:26:17.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:16 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:17.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:16 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:26:17.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:16 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:26:17.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:16 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:26:17.148 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:16 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:17.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:16 smithi060 ceph-mon[96524]: from='osd.9 [v2:172.21.15.60:6826/634188302,v1:172.21.15.60:6827/634188302]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2023-07-20T01:26:17.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:16 smithi060 ceph-mon[96524]: osdmap e64: 10 total, 9 up, 10 in 2023-07-20T01:26:17.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:16 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T01:26:17.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:16 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T01:26:17.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:16 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:17.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:16 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:17.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:16 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:26:17.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:16 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:26:17.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:16 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:26:17.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:16 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:26:17.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:16 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:17.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:16 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:26:17.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:16 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:26:17.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:16 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:17.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:16 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:26:17.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:16 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:26:17.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:16 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:26:17.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:16 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:17.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:16 smithi183 ceph-mon[98477]: from='osd.9 [v2:172.21.15.60:6826/634188302,v1:172.21.15.60:6827/634188302]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2023-07-20T01:26:17.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:16 smithi183 ceph-mon[98477]: osdmap e64: 10 total, 9 up, 10 in 2023-07-20T01:26:17.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:16 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T01:26:17.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:16 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T01:26:17.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:16 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:17.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:16 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:17.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:16 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:26:17.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:16 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:26:17.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:16 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:26:17.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:16 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:26:17.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:16 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:17.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:16 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:26:17.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:16 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:26:17.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:16 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:17.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:16 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:26:17.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:16 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:26:17.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:16 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:26:17.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:16 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:17.293 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.b/config 2023-07-20T01:26:17.803 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:17 smithi136 ceph-mon[98083]: purged_snaps scrub starts 2023-07-20T01:26:17.803 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:17 smithi136 ceph-mon[98083]: purged_snaps scrub ok 2023-07-20T01:26:17.803 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:17 smithi136 ceph-mon[98083]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T01:26:17.804 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:17 smithi136 ceph-mon[98083]: Detected new or changed devices on smithi060 2023-07-20T01:26:17.804 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:17 smithi136 ceph-mon[98083]: Adjusting osd_memory_target on smithi060 to 4293M 2023-07-20T01:26:17.804 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:17 smithi136 ceph-mon[98083]: osd.9 [v2:172.21.15.60:6826/634188302,v1:172.21.15.60:6827/634188302] boot 2023-07-20T01:26:17.804 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:17 smithi136 ceph-mon[98083]: osdmap e65: 10 total, 10 up, 10 in 2023-07-20T01:26:17.804 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:17 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T01:26:18.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:17 smithi060 ceph-mon[96524]: purged_snaps scrub starts 2023-07-20T01:26:18.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:17 smithi060 ceph-mon[96524]: purged_snaps scrub ok 2023-07-20T01:26:18.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:17 smithi060 ceph-mon[96524]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T01:26:18.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:17 smithi060 ceph-mon[96524]: Detected new or changed devices on smithi060 2023-07-20T01:26:18.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:17 smithi060 ceph-mon[96524]: Adjusting osd_memory_target on smithi060 to 4293M 2023-07-20T01:26:18.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:17 smithi060 ceph-mon[96524]: osd.9 [v2:172.21.15.60:6826/634188302,v1:172.21.15.60:6827/634188302] boot 2023-07-20T01:26:18.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:17 smithi060 ceph-mon[96524]: osdmap e65: 10 total, 10 up, 10 in 2023-07-20T01:26:18.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:17 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T01:26:18.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:17 smithi183 ceph-mon[98477]: purged_snaps scrub starts 2023-07-20T01:26:18.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:17 smithi183 ceph-mon[98477]: purged_snaps scrub ok 2023-07-20T01:26:18.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:17 smithi183 ceph-mon[98477]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-20T01:26:18.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:17 smithi183 ceph-mon[98477]: Detected new or changed devices on smithi060 2023-07-20T01:26:18.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:17 smithi183 ceph-mon[98477]: Adjusting osd_memory_target on smithi060 to 4293M 2023-07-20T01:26:18.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:17 smithi183 ceph-mon[98477]: osd.9 [v2:172.21.15.60:6826/634188302,v1:172.21.15.60:6827/634188302] boot 2023-07-20T01:26:18.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:17 smithi183 ceph-mon[98477]: osdmap e65: 10 total, 10 up, 10 in 2023-07-20T01:26:18.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:17 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-20T01:26:18.518 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.539+0000 7f1ddf98b700 1 -- 172.21.15.136:0/2979905872 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1dd81012a0 msgr2=0x7f1dd8107b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:26:18.519 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.539+0000 7f1ddf98b700 1 --2- 172.21.15.136:0/2979905872 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1dd81012a0 0x7f1dd8107b50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1dd000b0d0 tx=0x7f1dd000b3e0 comp rx=0 tx=0).stop 2023-07-20T01:26:18.519 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.540+0000 7f1ddf98b700 1 -- 172.21.15.136:0/2979905872 shutdown_connections 2023-07-20T01:26:18.519 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.540+0000 7f1ddf98b700 1 --2- 172.21.15.136:0/2979905872 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1dd81012a0 0x7f1dd8107b50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:26:18.519 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.540+0000 7f1ddf98b700 1 --2- 172.21.15.136:0/2979905872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1dd80ff700 0x7f1dd80ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:26:18.519 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.540+0000 7f1ddf98b700 1 --2- 172.21.15.136:0/2979905872 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1dd8100900 0x7f1dd8100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:26:18.519 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.540+0000 7f1ddf98b700 1 -- 172.21.15.136:0/2979905872 >> 172.21.15.136:0/2979905872 conn(0x7f1dd80fac80 msgr2=0x7f1dd80fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:26:18.520 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.540+0000 7f1ddf98b700 1 -- 172.21.15.136:0/2979905872 shutdown_connections 2023-07-20T01:26:18.520 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.540+0000 7f1ddf98b700 1 -- 172.21.15.136:0/2979905872 wait complete. 2023-07-20T01:26:18.520 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.541+0000 7f1ddf98b700 1 Processor -- start 2023-07-20T01:26:18.520 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.541+0000 7f1ddf98b700 1 -- start start 2023-07-20T01:26:18.521 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.541+0000 7f1ddf98b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1dd80ff700 0x7f1dd8103690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:26:18.522 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.541+0000 7f1ddf98b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1dd8100900 0x7f1dd8101ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:26:18.522 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.541+0000 7f1ddf98b700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1dd81012a0 0x7f1dd8102220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:26:18.522 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.541+0000 7f1ddf98b700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd81029a0 con 0x7f1dd8100900 2023-07-20T01:26:18.522 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.541+0000 7f1ddf98b700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd8102b10 con 0x7f1dd80ff700 2023-07-20T01:26:18.522 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.541+0000 7f1ddf98b700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd81a0550 con 0x7f1dd81012a0 2023-07-20T01:26:18.522 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.541+0000 7f1ddd727700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1dd80ff700 0x7f1dd8103690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:26:18.523 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.541+0000 7f1ddd727700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1dd80ff700 0x7f1dd8103690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38274/0 (socket says 172.21.15.136:38274) 2023-07-20T01:26:18.523 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.541+0000 7f1ddd727700 1 -- 172.21.15.136:0/2029279565 learned_addr learned my addr 172.21.15.136:0/2029279565 (peer_addr_for_me v2:172.21.15.136:0/0) 2023-07-20T01:26:18.523 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.541+0000 7f1dddf28700 1 --2- 172.21.15.136:0/2029279565 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1dd81012a0 0x7f1dd8102220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:26:18.523 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.541+0000 7f1ddcf26700 1 --2- 172.21.15.136:0/2029279565 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1dd8100900 0x7f1dd8101ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:26:18.523 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.541+0000 7f1ddd727700 1 -- 172.21.15.136:0/2029279565 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1dd81012a0 msgr2=0x7f1dd8102220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:26:18.523 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.541+0000 7f1ddd727700 1 --2- 172.21.15.136:0/2029279565 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1dd81012a0 0x7f1dd8102220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:26:18.523 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.541+0000 7f1ddd727700 1 -- 172.21.15.136:0/2029279565 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1dd8100900 msgr2=0x7f1dd8101ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:26:18.524 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.541+0000 7f1ddd727700 1 --2- 172.21.15.136:0/2029279565 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1dd8100900 0x7f1dd8101ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:26:18.524 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.541+0000 7f1ddd727700 1 -- 172.21.15.136:0/2029279565 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd0009d00 con 0x7f1dd80ff700 2023-07-20T01:26:18.524 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.542+0000 7f1ddd727700 1 --2- 172.21.15.136:0/2029279565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1dd80ff700 0x7f1dd8103690 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc00c970 tx=0x7f1dcc00cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:26:18.524 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.542+0000 7f1dca7fc700 1 -- 172.21.15.136:0/2029279565 <== mon.2 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc004220 con 0x7f1dd80ff700 2023-07-20T01:26:18.524 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.542+0000 7f1dca7fc700 1 -- 172.21.15.136:0/2029279565 <== mon.2 v2:172.21.15.136:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1dcc007d30 con 0x7f1dd80ff700 2023-07-20T01:26:18.524 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.542+0000 7f1ddf98b700 1 -- 172.21.15.136:0/2029279565 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dd81a0830 con 0x7f1dd80ff700 2023-07-20T01:26:18.524 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.542+0000 7f1dca7fc700 1 -- 172.21.15.136:0/2029279565 <== mon.2 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc003b80 con 0x7f1dd80ff700 2023-07-20T01:26:18.525 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.542+0000 7f1ddf98b700 1 -- 172.21.15.136:0/2029279565 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dd81a0d30 con 0x7f1dd80ff700 2023-07-20T01:26:18.525 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.543+0000 7f1ddf98b700 1 -- 172.21.15.136:0/2029279565 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dd810ade0 con 0x7f1dd80ff700 2023-07-20T01:26:18.525 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.545+0000 7f1dca7fc700 1 -- 172.21.15.136:0/2029279565 <== mon.2 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f1dcc003ce0 con 0x7f1dd80ff700 2023-07-20T01:26:18.526 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.545+0000 7f1dca7fc700 1 --2- 172.21.15.136:0/2029279565 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f1dc40a1150 0x7f1dc40a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:26:18.526 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.545+0000 7f1dca7fc700 1 -- 172.21.15.136:0/2029279565 <== mon.2 v2:172.21.15.136:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f1dcc0cc0e0 con 0x7f1dd80ff700 2023-07-20T01:26:18.526 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.545+0000 7f1ddcf26700 1 --2- 172.21.15.136:0/2029279565 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f1dc40a1150 0x7f1dc40a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:26:18.526 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.546+0000 7f1ddcf26700 1 --2- 172.21.15.136:0/2029279565 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f1dc40a1150 0x7f1dc40a3610 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4005950 tx=0x7f1dd40058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:26:18.526 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.549+0000 7f1dca7fc700 1 -- 172.21.15.136:0/2029279565 <== mon.2 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1dcc096030 con 0x7f1dd80ff700 2023-07-20T01:26:18.719 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:18.742+0000 7f1ddf98b700 1 -- 172.21.15.136:0/2029279565 --> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f1dd80689c0 con 0x7f1dc40a1150 2023-07-20T01:26:19.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:18 smithi136 ceph-mon[98083]: osdmap e66: 10 total, 10 up, 10 in 2023-07-20T01:26:19.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:18 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:26:19.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:18 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:26:19.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:18 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:26:19.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:18 smithi060 ceph-mon[96524]: osdmap e66: 10 total, 10 up, 10 in 2023-07-20T01:26:19.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:18 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:26:19.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:18 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:26:19.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:18 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:26:19.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:18 smithi183 ceph-mon[98477]: osdmap e66: 10 total, 10 up, 10 in 2023-07-20T01:26:19.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:18 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:26:19.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:18 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:26:19.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:18 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:26:19.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:19 smithi136 ceph-mon[98083]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:19.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:19 smithi136 ceph-mon[98083]: from='client.24359 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:26:20.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:19 smithi060 ceph-mon[96524]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:20.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:19 smithi060 ceph-mon[96524]: from='client.24359 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:26:20.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:19 smithi183 ceph-mon[98477]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:20.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:19 smithi183 ceph-mon[98477]: from='client.24359 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:26:22.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:21 smithi136 ceph-mon[98083]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:22.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:21 smithi136 ceph-mon[98083]: from='client.? 172.21.15.136:0/137609754' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b092b5a-8512-47d5-928b-995f8dc17353"}]: dispatch 2023-07-20T01:26:22.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:21 smithi136 ceph-mon[98083]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b092b5a-8512-47d5-928b-995f8dc17353"}]: dispatch 2023-07-20T01:26:22.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:21 smithi136 ceph-mon[98083]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b092b5a-8512-47d5-928b-995f8dc17353"}]': finished 2023-07-20T01:26:22.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:21 smithi136 ceph-mon[98083]: osdmap e67: 11 total, 10 up, 11 in 2023-07-20T01:26:22.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:21 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T01:26:22.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:21 smithi060 ceph-mon[96524]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:22.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:21 smithi060 ceph-mon[96524]: from='client.? 172.21.15.136:0/137609754' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b092b5a-8512-47d5-928b-995f8dc17353"}]: dispatch 2023-07-20T01:26:22.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:21 smithi060 ceph-mon[96524]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b092b5a-8512-47d5-928b-995f8dc17353"}]: dispatch 2023-07-20T01:26:22.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:21 smithi060 ceph-mon[96524]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b092b5a-8512-47d5-928b-995f8dc17353"}]': finished 2023-07-20T01:26:22.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:21 smithi060 ceph-mon[96524]: osdmap e67: 11 total, 10 up, 11 in 2023-07-20T01:26:22.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:21 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T01:26:22.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:21 smithi183 ceph-mon[98477]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:22.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:21 smithi183 ceph-mon[98477]: from='client.? 172.21.15.136:0/137609754' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b092b5a-8512-47d5-928b-995f8dc17353"}]: dispatch 2023-07-20T01:26:22.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:21 smithi183 ceph-mon[98477]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b092b5a-8512-47d5-928b-995f8dc17353"}]: dispatch 2023-07-20T01:26:22.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:21 smithi183 ceph-mon[98477]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b092b5a-8512-47d5-928b-995f8dc17353"}]': finished 2023-07-20T01:26:22.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:21 smithi183 ceph-mon[98477]: osdmap e67: 11 total, 10 up, 11 in 2023-07-20T01:26:22.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:21 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T01:26:23.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:22 smithi136 ceph-mon[98083]: from='client.? 172.21.15.136:0/1089697968' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:26:23.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:22 smithi060 ceph-mon[96524]: from='client.? 172.21.15.136:0/1089697968' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:26:23.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:22 smithi183 ceph-mon[98477]: from='client.? 172.21.15.136:0/1089697968' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:26:24.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:23 smithi136 ceph-mon[98083]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:24.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:23 smithi060 ceph-mon[96524]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:24.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:23 smithi183 ceph-mon[98477]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:26.065 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:25 smithi136 ceph-mon[98083]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:26.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:25 smithi060 ceph-mon[96524]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:26.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:25 smithi183 ceph-mon[98477]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:28.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:27 smithi136 ceph-mon[98083]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:28.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:27 smithi060 ceph-mon[96524]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:28.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:27 smithi183 ceph-mon[98477]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:29.141 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:28 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-20T01:26:29.141 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:28 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:26:29.141 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:28 smithi136 ceph-mon[98083]: Deploying daemon osd.10 on smithi136 2023-07-20T01:26:29.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:28 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-20T01:26:29.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:28 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:26:29.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:28 smithi060 ceph-mon[96524]: Deploying daemon osd.10 on smithi136 2023-07-20T01:26:29.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:28 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-20T01:26:29.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:28 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:26:29.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:28 smithi183 ceph-mon[98477]: Deploying daemon osd.10 on smithi136 2023-07-20T01:26:30.062 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:29 smithi136 ceph-mon[98083]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:30.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:29 smithi060 ceph-mon[96524]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:30.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:29 smithi183 ceph-mon[98477]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:32.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:31 smithi136 ceph-mon[98083]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:32.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:31 smithi060 ceph-mon[96524]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:32.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:31 smithi183 ceph-mon[98477]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:33.832 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:33 smithi136 ceph-mon[98083]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:33.833 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:33 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:33.833 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:33 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:33.833 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:33 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:26:33.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:33 smithi060 ceph-mon[96524]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:33.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:33 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:33.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:33 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:33.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:33 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:26:33.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:33 smithi183 ceph-mon[98477]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:33.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:33 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:33.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:33 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:33.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:33 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:26:35.632 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:35.654+0000 7f1dca7fc700 1 -- 172.21.15.136:0/2029279565 <== mgr.14152 v2:172.21.15.60:6800/1961833180 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f1dd80689c0 con 0x7f1dc40a1150 2023-07-20T01:26:35.633 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 10 on host 'smithi136' 2023-07-20T01:26:35.636 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:35.657+0000 7f1ddf98b700 1 -- 172.21.15.136:0/2029279565 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f1dc40a1150 msgr2=0x7f1dc40a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:26:35.636 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:35.657+0000 7f1ddf98b700 1 --2- 172.21.15.136:0/2029279565 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f1dc40a1150 0x7f1dc40a3610 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4005950 tx=0x7f1dd40058e0 comp rx=0 tx=0).stop 2023-07-20T01:26:35.636 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:35.657+0000 7f1ddf98b700 1 -- 172.21.15.136:0/2029279565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1dd80ff700 msgr2=0x7f1dd8103690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:26:35.636 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:35.657+0000 7f1ddf98b700 1 --2- 172.21.15.136:0/2029279565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1dd80ff700 0x7f1dd8103690 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc00c970 tx=0x7f1dcc00cc80 comp rx=0 tx=0).stop 2023-07-20T01:26:35.636 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:35.658+0000 7f1ddf98b700 1 -- 172.21.15.136:0/2029279565 shutdown_connections 2023-07-20T01:26:35.637 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:35.658+0000 7f1ddf98b700 1 --2- 172.21.15.136:0/2029279565 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f1dc40a1150 0x7f1dc40a3610 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:26:35.637 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:35.658+0000 7f1ddf98b700 1 --2- 172.21.15.136:0/2029279565 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1dd81012a0 0x7f1dd8102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:26:35.637 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:35.658+0000 7f1ddf98b700 1 --2- 172.21.15.136:0/2029279565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1dd80ff700 0x7f1dd8103690 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:26:35.637 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:35.658+0000 7f1ddf98b700 1 --2- 172.21.15.136:0/2029279565 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1dd8100900 0x7f1dd8101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:26:35.637 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:35.658+0000 7f1ddf98b700 1 -- 172.21.15.136:0/2029279565 >> 172.21.15.136:0/2029279565 conn(0x7f1dd80fac80 msgr2=0x7f1dd80682b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:26:35.637 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:35.658+0000 7f1ddf98b700 1 -- 172.21.15.136:0/2029279565 shutdown_connections 2023-07-20T01:26:35.637 INFO:teuthology.orchestra.run.smithi136.stderr:2023-07-20T01:26:35.658+0000 7f1ddf98b700 1 -- 172.21.15.136:0/2029279565 wait complete. 2023-07-20T01:26:35.823 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:35 smithi136 ceph-mon[98083]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:35.824 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:35 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:35.824 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:35 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:35.824 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:35 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:26:35.824 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:35 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:26:35.824 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:35 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:35.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:35 smithi060 ceph-mon[96524]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:35.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:35 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:35.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:35 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:35.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:35 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:26:35.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:35 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:26:35.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:35 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:35.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:35 smithi183 ceph-mon[98477]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:35.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:35 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:35.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:35 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:35.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:35 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:26:35.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:35 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:26:35.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:35 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:36.192 DEBUG:teuthology.orchestra.run.smithi136:osd.10> sudo journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.10.service 2023-07-20T01:26:36.194 INFO:tasks.cephadm:Deploying osd.11 on smithi183 with /dev/vg_nvme/lv_1... 2023-07-20T01:26:36.194 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-20T01:26:36.223 INFO:journalctl@ceph.osd.10.smithi136.stdout:-- Logs begin at Thu 2023-07-20 01:00:24 UTC. -- 2023-07-20T01:26:36.619 INFO:journalctl@ceph.osd.10.smithi136.stdout:Jul 20 01:26:36 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-10[119505]: 2023-07-20T01:26:36.437+0000 7fbf4e0cb640 -1 osd.10 0 log_to_monitors true 2023-07-20T01:26:36.758 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T01:26:36.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:36 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:36.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:36 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:36.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:36 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:26:36.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:36 smithi136 ceph-mon[98083]: from='osd.10 [v2:172.21.15.136:6824/247446439,v1:172.21.15.136:6825/247446439]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T01:26:36.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:36 smithi136 ceph-mon[98083]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T01:26:36.924 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:36 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:36.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:36 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:36.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:36 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:26:36.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:36 smithi060 ceph-mon[96524]: from='osd.10 [v2:172.21.15.136:6824/247446439,v1:172.21.15.136:6825/247446439]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T01:26:36.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:36 smithi060 ceph-mon[96524]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T01:26:36.947 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:36 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:36.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:36 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:36.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:36 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:26:36.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:36 smithi183 ceph-mon[98477]: from='osd.10 [v2:172.21.15.136:6824/247446439,v1:172.21.15.136:6825/247446439]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T01:26:36.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:36 smithi183 ceph-mon[98477]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-20T01:26:37.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:37 smithi136 ceph-mon[98083]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:37.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:37 smithi136 ceph-mon[98083]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-20T01:26:37.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:37 smithi136 ceph-mon[98083]: osdmap e68: 11 total, 10 up, 11 in 2023-07-20T01:26:37.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:37 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T01:26:37.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:37 smithi136 ceph-mon[98083]: from='osd.10 [v2:172.21.15.136:6824/247446439,v1:172.21.15.136:6825/247446439]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-07-20T01:26:37.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:37 smithi136 ceph-mon[98083]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-07-20T01:26:37.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:37 smithi060 ceph-mon[96524]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:37.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:37 smithi060 ceph-mon[96524]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-20T01:26:37.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:37 smithi060 ceph-mon[96524]: osdmap e68: 11 total, 10 up, 11 in 2023-07-20T01:26:37.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:37 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T01:26:37.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:37 smithi060 ceph-mon[96524]: from='osd.10 [v2:172.21.15.136:6824/247446439,v1:172.21.15.136:6825/247446439]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-07-20T01:26:37.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:37 smithi060 ceph-mon[96524]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-07-20T01:26:37.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:37 smithi183 ceph-mon[98477]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:37.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:37 smithi183 ceph-mon[98477]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-20T01:26:37.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:37 smithi183 ceph-mon[98477]: osdmap e68: 11 total, 10 up, 11 in 2023-07-20T01:26:37.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:37 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T01:26:37.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:37 smithi183 ceph-mon[98477]: from='osd.10 [v2:172.21.15.136:6824/247446439,v1:172.21.15.136:6825/247446439]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-07-20T01:26:37.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:37 smithi183 ceph-mon[98477]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-07-20T01:26:38.562 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-20T01:26:38.562 INFO:teuthology.orchestra.run.smithi183.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-20T01:26:38.562 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10+0 records in 2023-07-20T01:26:38.563 INFO:teuthology.orchestra.run.smithi183.stderr:10+0 records out 2023-07-20T01:26:38.563 INFO:teuthology.orchestra.run.smithi183.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0156568 s, 670 MB/s 2023-07-20T01:26:38.563 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping successful for: 2023-07-20T01:26:38.567 INFO:journalctl@ceph.osd.10.smithi136.stdout:Jul 20 01:26:38 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-10[119505]: 2023-07-20T01:26:38.392+0000 7fbf43743700 -1 osd.10 0 waiting for initial osdmap 2023-07-20T01:26:38.568 INFO:journalctl@ceph.osd.10.smithi136.stdout:Jul 20 01:26:38 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-10[119505]: 2023-07-20T01:26:38.403+0000 7fbf3d535700 -1 osd.10 69 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T01:26:39.000 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph orch daemon add osd smithi183:vg_nvme/lv_1 2023-07-20T01:26:39.548 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T01:26:39.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:39 smithi136 ceph-mon[98083]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:39.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:39 smithi136 ceph-mon[98083]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2023-07-20T01:26:39.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:39 smithi136 ceph-mon[98083]: osdmap e69: 11 total, 10 up, 11 in 2023-07-20T01:26:39.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:39 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T01:26:39.646 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:39 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T01:26:39.646 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:39 smithi136 ceph-mon[98083]: Detected new or changed devices on smithi136 2023-07-20T01:26:39.646 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:39 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:39.646 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:39 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:39.646 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:39 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:26:39.646 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:39 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:26:39.646 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:39 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:26:39.646 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:39 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:26:39.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:39 smithi136 ceph-mon[98083]: Adjusting osd_memory_target on smithi136 to 4292M 2023-07-20T01:26:39.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:39 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:39.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:39 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:26:39.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:39 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:26:39.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:39 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:39.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:39 smithi060 ceph-mon[96524]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:39.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:39 smithi060 ceph-mon[96524]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2023-07-20T01:26:39.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:39 smithi060 ceph-mon[96524]: osdmap e69: 11 total, 10 up, 11 in 2023-07-20T01:26:39.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:39 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T01:26:39.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:39 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T01:26:39.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:39 smithi060 ceph-mon[96524]: Detected new or changed devices on smithi136 2023-07-20T01:26:39.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:39 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:39.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:39 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:39.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:39 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:26:39.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:39 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:26:39.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:39 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:26:39.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:39 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:26:39.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:39 smithi060 ceph-mon[96524]: Adjusting osd_memory_target on smithi136 to 4292M 2023-07-20T01:26:39.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:39 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:39.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:39 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:26:39.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:39 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:26:39.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:39 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:39.697 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:39 smithi183 ceph-mon[98477]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-20T01:26:39.698 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:39 smithi183 ceph-mon[98477]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2023-07-20T01:26:39.698 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:39 smithi183 ceph-mon[98477]: osdmap e69: 11 total, 10 up, 11 in 2023-07-20T01:26:39.698 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:39 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T01:26:39.698 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:39 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T01:26:39.698 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:39 smithi183 ceph-mon[98477]: Detected new or changed devices on smithi136 2023-07-20T01:26:39.698 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:39 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:39.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:39 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:39.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:39 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:26:39.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:39 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:26:39.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:39 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:26:39.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:39 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:26:39.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:39 smithi183 ceph-mon[98477]: Adjusting osd_memory_target on smithi136 to 4292M 2023-07-20T01:26:39.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:39 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:39.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:39 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:26:39.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:39 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:26:39.700 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:39 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:40.638 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:40 smithi183 ceph-mon[98477]: purged_snaps scrub starts 2023-07-20T01:26:40.638 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:40 smithi183 ceph-mon[98477]: purged_snaps scrub ok 2023-07-20T01:26:40.639 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:40 smithi183 ceph-mon[98477]: osd.10 [v2:172.21.15.136:6824/247446439,v1:172.21.15.136:6825/247446439] boot 2023-07-20T01:26:40.639 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:40 smithi183 ceph-mon[98477]: osdmap e70: 11 total, 11 up, 11 in 2023-07-20T01:26:40.639 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:40 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T01:26:40.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:40 smithi136 ceph-mon[98083]: purged_snaps scrub starts 2023-07-20T01:26:40.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:40 smithi136 ceph-mon[98083]: purged_snaps scrub ok 2023-07-20T01:26:40.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:40 smithi136 ceph-mon[98083]: osd.10 [v2:172.21.15.136:6824/247446439,v1:172.21.15.136:6825/247446439] boot 2023-07-20T01:26:40.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:40 smithi136 ceph-mon[98083]: osdmap e70: 11 total, 11 up, 11 in 2023-07-20T01:26:40.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:40 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T01:26:40.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:40 smithi060 ceph-mon[96524]: purged_snaps scrub starts 2023-07-20T01:26:40.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:40 smithi060 ceph-mon[96524]: purged_snaps scrub ok 2023-07-20T01:26:40.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:40 smithi060 ceph-mon[96524]: osd.10 [v2:172.21.15.136:6824/247446439,v1:172.21.15.136:6825/247446439] boot 2023-07-20T01:26:40.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:40 smithi060 ceph-mon[96524]: osdmap e70: 11 total, 11 up, 11 in 2023-07-20T01:26:40.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:40 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-20T01:26:40.820 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.840+0000 7fe59a6c1700 1 -- 172.21.15.183:0/1530297337 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe5941012a0 msgr2=0x7fe594107b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:26:40.820 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.840+0000 7fe59a6c1700 1 --2- 172.21.15.183:0/1530297337 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe5941012a0 0x7fe594107b50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe58c00b0a0 tx=0x7fe58c00b3b0 comp rx=0 tx=0).stop 2023-07-20T01:26:40.820 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.840+0000 7fe59a6c1700 1 -- 172.21.15.183:0/1530297337 shutdown_connections 2023-07-20T01:26:40.820 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.840+0000 7fe59a6c1700 1 --2- 172.21.15.183:0/1530297337 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe5941012a0 0x7fe594107b50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:26:40.821 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.840+0000 7fe59a6c1700 1 --2- 172.21.15.183:0/1530297337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5940ff700 0x7fe5940ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:26:40.821 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.840+0000 7fe59a6c1700 1 --2- 172.21.15.183:0/1530297337 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe594100900 0x7fe594100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:26:40.821 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.840+0000 7fe59a6c1700 1 -- 172.21.15.183:0/1530297337 >> 172.21.15.183:0/1530297337 conn(0x7fe5940fac80 msgr2=0x7fe5940fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:26:40.821 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.840+0000 7fe59a6c1700 1 -- 172.21.15.183:0/1530297337 shutdown_connections 2023-07-20T01:26:40.821 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.840+0000 7fe59a6c1700 1 -- 172.21.15.183:0/1530297337 wait complete. 2023-07-20T01:26:40.821 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.841+0000 7fe59a6c1700 1 Processor -- start 2023-07-20T01:26:40.821 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.841+0000 7fe59a6c1700 1 -- start start 2023-07-20T01:26:40.822 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.841+0000 7fe59a6c1700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5940ff700 0x7fe594068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:26:40.822 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.841+0000 7fe59a6c1700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe594100900 0x7fe59419b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:26:40.822 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.841+0000 7fe59a6c1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5941012a0 0x7fe59419d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:26:40.822 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.841+0000 7fe59a6c1700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59419e090 con 0x7fe5941012a0 2023-07-20T01:26:40.822 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.841+0000 7fe59a6c1700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59419e200 con 0x7fe5940ff700 2023-07-20T01:26:40.822 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.841+0000 7fe59a6c1700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59419e3c0 con 0x7fe594100900 2023-07-20T01:26:40.822 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.841+0000 7fe5937fe700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe594100900 0x7fe59419b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:26:40.823 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.841+0000 7fe5937fe700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe594100900 0x7fe59419b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:46210/0 (socket says 172.21.15.183:46210) 2023-07-20T01:26:40.823 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.841+0000 7fe5937fe700 1 -- 172.21.15.183:0/4009085047 learned_addr learned my addr 172.21.15.183:0/4009085047 (peer_addr_for_me v2:172.21.15.183:0/0) 2023-07-20T01:26:40.823 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.841+0000 7fe598c5e700 1 --2- 172.21.15.183:0/4009085047 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5941012a0 0x7fe59419d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:26:40.823 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.841+0000 7fe593fff700 1 --2- 172.21.15.183:0/4009085047 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5940ff700 0x7fe594068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:26:40.823 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.842+0000 7fe5937fe700 1 -- 172.21.15.183:0/4009085047 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5940ff700 msgr2=0x7fe594068d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:26:40.823 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.842+0000 7fe5937fe700 1 --2- 172.21.15.183:0/4009085047 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5940ff700 0x7fe594068d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:26:40.823 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.842+0000 7fe5937fe700 1 -- 172.21.15.183:0/4009085047 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5941012a0 msgr2=0x7fe59419d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:26:40.823 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.842+0000 7fe5937fe700 1 --2- 172.21.15.183:0/4009085047 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5941012a0 0x7fe59419d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:26:40.824 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.842+0000 7fe5937fe700 1 -- 172.21.15.183:0/4009085047 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe58c009d00 con 0x7fe594100900 2023-07-20T01:26:40.824 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.842+0000 7fe5937fe700 1 --2- 172.21.15.183:0/4009085047 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe594100900 0x7fe59419b2b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe57c00b530 tx=0x7fe57c00b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:26:40.824 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.842+0000 7fe5917fa700 1 -- 172.21.15.183:0/4009085047 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57c011830 con 0x7fe594100900 2023-07-20T01:26:40.824 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.842+0000 7fe59a6c1700 1 -- 172.21.15.183:0/4009085047 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe59419e6a0 con 0x7fe594100900 2023-07-20T01:26:40.824 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.842+0000 7fe5917fa700 1 -- 172.21.15.183:0/4009085047 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe57c011e20 con 0x7fe594100900 2023-07-20T01:26:40.824 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.842+0000 7fe5917fa700 1 -- 172.21.15.183:0/4009085047 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57c010db0 con 0x7fe594100900 2023-07-20T01:26:40.824 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.842+0000 7fe59a6c1700 1 -- 172.21.15.183:0/4009085047 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe59419ec20 con 0x7fe594100900 2023-07-20T01:26:40.825 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.843+0000 7fe59a6c1700 1 -- 172.21.15.183:0/4009085047 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe594106d50 con 0x7fe594100900 2023-07-20T01:26:40.825 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.844+0000 7fe5917fa700 1 -- 172.21.15.183:0/4009085047 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe57c011990 con 0x7fe594100900 2023-07-20T01:26:40.826 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.845+0000 7fe5917fa700 1 --2- 172.21.15.183:0/4009085047 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fe5800a11d0 0x7fe5800a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:26:40.826 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.845+0000 7fe5917fa700 1 -- 172.21.15.183:0/4009085047 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6723+0+0 (secure 0 0 0) 0x7fe57c0cc170 con 0x7fe594100900 2023-07-20T01:26:40.826 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.845+0000 7fe593fff700 1 --2- 172.21.15.183:0/4009085047 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fe5800a11d0 0x7fe5800a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:26:40.826 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.845+0000 7fe593fff700 1 --2- 172.21.15.183:0/4009085047 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fe5800a11d0 0x7fe5800a3690 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe594100760 tx=0x7fe58400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:26:40.826 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:40.848+0000 7fe5917fa700 1 -- 172.21.15.183:0/4009085047 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe57c095e50 con 0x7fe594100900 2023-07-20T01:26:41.020 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:41.042+0000 7fe59a6c1700 1 -- 172.21.15.183:0/4009085047 --> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe594060bb0 con 0x7fe5800a11d0 2023-07-20T01:26:41.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:41 smithi136 ceph-mon[98083]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:26:41.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:41 smithi136 ceph-mon[98083]: osdmap e71: 11 total, 11 up, 11 in 2023-07-20T01:26:41.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:41 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:26:41.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:41 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:26:41.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:41 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:26:41.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:41 smithi060 ceph-mon[96524]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:26:41.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:41 smithi060 ceph-mon[96524]: osdmap e71: 11 total, 11 up, 11 in 2023-07-20T01:26:41.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:41 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:26:41.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:41 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:26:41.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:41 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:26:41.697 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:41 smithi183 ceph-mon[98477]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:26:41.698 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:41 smithi183 ceph-mon[98477]: osdmap e71: 11 total, 11 up, 11 in 2023-07-20T01:26:41.698 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:41 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-20T01:26:41.698 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:41 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-20T01:26:41.698 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:41 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:26:42.618 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:42 smithi183 ceph-mon[98477]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:26:42.618 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:42 smithi183 ceph-mon[98477]: osdmap e72: 11 total, 11 up, 11 in 2023-07-20T01:26:42.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:42 smithi136 ceph-mon[98083]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:26:42.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:42 smithi136 ceph-mon[98083]: osdmap e72: 11 total, 11 up, 11 in 2023-07-20T01:26:42.674 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:42 smithi060 ceph-mon[96524]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:26:42.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:42 smithi060 ceph-mon[96524]: osdmap e72: 11 total, 11 up, 11 in 2023-07-20T01:26:43.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:43 smithi136 ceph-mon[98083]: pgmap v214: 1 pgs: 1 remapped+peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:26:43.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:43 smithi136 ceph-mon[98083]: osdmap e73: 11 total, 11 up, 11 in 2023-07-20T01:26:43.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:43 smithi136 ceph-mon[98083]: from='client.? 172.21.15.183:0/60235916' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8415b1a9-eb18-4b27-ad4f-9a676757bef9"}]: dispatch 2023-07-20T01:26:43.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:43 smithi136 ceph-mon[98083]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8415b1a9-eb18-4b27-ad4f-9a676757bef9"}]: dispatch 2023-07-20T01:26:43.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:43 smithi136 ceph-mon[98083]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8415b1a9-eb18-4b27-ad4f-9a676757bef9"}]': finished 2023-07-20T01:26:43.646 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:43 smithi136 ceph-mon[98083]: osdmap e74: 12 total, 11 up, 12 in 2023-07-20T01:26:43.646 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:43 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T01:26:43.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:43 smithi060 ceph-mon[96524]: pgmap v214: 1 pgs: 1 remapped+peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:26:43.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:43 smithi060 ceph-mon[96524]: osdmap e73: 11 total, 11 up, 11 in 2023-07-20T01:26:43.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:43 smithi060 ceph-mon[96524]: from='client.? 172.21.15.183:0/60235916' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8415b1a9-eb18-4b27-ad4f-9a676757bef9"}]: dispatch 2023-07-20T01:26:43.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:43 smithi060 ceph-mon[96524]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8415b1a9-eb18-4b27-ad4f-9a676757bef9"}]: dispatch 2023-07-20T01:26:43.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:43 smithi060 ceph-mon[96524]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8415b1a9-eb18-4b27-ad4f-9a676757bef9"}]': finished 2023-07-20T01:26:43.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:43 smithi060 ceph-mon[96524]: osdmap e74: 12 total, 11 up, 12 in 2023-07-20T01:26:43.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:43 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T01:26:43.698 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:43 smithi183 ceph-mon[98477]: pgmap v214: 1 pgs: 1 remapped+peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:26:43.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:43 smithi183 ceph-mon[98477]: osdmap e73: 11 total, 11 up, 11 in 2023-07-20T01:26:43.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:43 smithi183 ceph-mon[98477]: from='client.? 172.21.15.183:0/60235916' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8415b1a9-eb18-4b27-ad4f-9a676757bef9"}]: dispatch 2023-07-20T01:26:43.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:43 smithi183 ceph-mon[98477]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8415b1a9-eb18-4b27-ad4f-9a676757bef9"}]: dispatch 2023-07-20T01:26:43.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:43 smithi183 ceph-mon[98477]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8415b1a9-eb18-4b27-ad4f-9a676757bef9"}]': finished 2023-07-20T01:26:43.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:43 smithi183 ceph-mon[98477]: osdmap e74: 12 total, 11 up, 12 in 2023-07-20T01:26:43.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:43 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T01:26:44.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:44 smithi136 ceph-mon[98083]: from='client.? 172.21.15.183:0/1194830233' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:26:44.674 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:44 smithi060 ceph-mon[96524]: from='client.? 172.21.15.183:0/1194830233' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:26:44.697 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:44 smithi183 ceph-mon[98477]: from='client.? 172.21.15.183:0/1194830233' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-20T01:26:45.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:45 smithi136 ceph-mon[98083]: pgmap v217: 1 pgs: 1 remapped+peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:26:45.674 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:45 smithi060 ceph-mon[96524]: pgmap v217: 1 pgs: 1 remapped+peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:26:45.697 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:45 smithi183 ceph-mon[98477]: pgmap v217: 1 pgs: 1 remapped+peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:26:47.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:47 smithi136 ceph-mon[98083]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 105 KiB/s, 0 objects/s recovering 2023-07-20T01:26:47.674 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:47 smithi060 ceph-mon[96524]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 105 KiB/s, 0 objects/s recovering 2023-07-20T01:26:47.697 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:47 smithi183 ceph-mon[98477]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 105 KiB/s, 0 objects/s recovering 2023-07-20T01:26:49.674 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:49 smithi060 ceph-mon[96524]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 89 KiB/s, 0 objects/s recovering 2023-07-20T01:26:49.697 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:49 smithi183 ceph-mon[98477]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 89 KiB/s, 0 objects/s recovering 2023-07-20T01:26:49.894 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:49 smithi136 ceph-mon[98083]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 89 KiB/s, 0 objects/s recovering 2023-07-20T01:26:50.576 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:50 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-20T01:26:50.576 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:50 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:26:50.577 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:50 smithi183 ceph-mon[98477]: Deploying daemon osd.11 on smithi183 2023-07-20T01:26:50.674 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:50 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-20T01:26:50.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:50 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:26:50.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:50 smithi060 ceph-mon[96524]: Deploying daemon osd.11 on smithi183 2023-07-20T01:26:50.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:50 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-20T01:26:50.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:50 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:26:50.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:50 smithi136 ceph-mon[98083]: Deploying daemon osd.11 on smithi183 2023-07-20T01:26:51.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:51 smithi060 ceph-mon[96524]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T01:26:51.698 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:51 smithi183 ceph-mon[98477]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T01:26:51.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:51 smithi136 ceph-mon[98083]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-20T01:26:53.674 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:53 smithi060 ceph-mon[96524]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-07-20T01:26:53.698 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:53 smithi183 ceph-mon[98477]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-07-20T01:26:53.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:53 smithi136 ceph-mon[98083]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-07-20T01:26:55.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:55 smithi183 ceph-mon[98477]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 55 KiB/s, 0 objects/s recovering 2023-07-20T01:26:55.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:55 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:55.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:55 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:55.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:55 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:26:55.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:55 smithi136 ceph-mon[98083]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 55 KiB/s, 0 objects/s recovering 2023-07-20T01:26:55.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:55 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:55.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:55 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:55.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:55 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:26:55.424 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:55 smithi060 ceph-mon[96524]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 55 KiB/s, 0 objects/s recovering 2023-07-20T01:26:55.425 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:55 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:55.425 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:55 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:55.425 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:55 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:26:56.992 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:57.014+0000 7fe5917fa700 1 -- 172.21.15.183:0/4009085047 <== mgr.14152 v2:172.21.15.60:6800/1961833180 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fe594060bb0 con 0x7fe5800a11d0 2023-07-20T01:26:56.992 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 11 on host 'smithi183' 2023-07-20T01:26:56.994 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:57.016+0000 7fe59a6c1700 1 -- 172.21.15.183:0/4009085047 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fe5800a11d0 msgr2=0x7fe5800a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:26:56.995 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:57.016+0000 7fe59a6c1700 1 --2- 172.21.15.183:0/4009085047 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fe5800a11d0 0x7fe5800a3690 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe594100760 tx=0x7fe58400b410 comp rx=0 tx=0).stop 2023-07-20T01:26:56.995 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:57.016+0000 7fe59a6c1700 1 -- 172.21.15.183:0/4009085047 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe594100900 msgr2=0x7fe59419b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:26:56.995 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:57.016+0000 7fe59a6c1700 1 --2- 172.21.15.183:0/4009085047 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe594100900 0x7fe59419b2b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe57c00b530 tx=0x7fe57c00b840 comp rx=0 tx=0).stop 2023-07-20T01:26:56.995 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:57.016+0000 7fe59a6c1700 1 -- 172.21.15.183:0/4009085047 shutdown_connections 2023-07-20T01:26:56.995 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:57.016+0000 7fe59a6c1700 1 --2- 172.21.15.183:0/4009085047 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fe5800a11d0 0x7fe5800a3690 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:26:56.995 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:57.016+0000 7fe59a6c1700 1 --2- 172.21.15.183:0/4009085047 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5941012a0 0x7fe59419d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:26:56.995 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:57.017+0000 7fe59a6c1700 1 --2- 172.21.15.183:0/4009085047 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5940ff700 0x7fe594068d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:26:56.996 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:57.017+0000 7fe59a6c1700 1 --2- 172.21.15.183:0/4009085047 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe594100900 0x7fe59419b2b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:26:56.996 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:57.017+0000 7fe59a6c1700 1 -- 172.21.15.183:0/4009085047 >> 172.21.15.183:0/4009085047 conn(0x7fe5940fac80 msgr2=0x7fe594103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:26:56.996 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:57.017+0000 7fe59a6c1700 1 -- 172.21.15.183:0/4009085047 shutdown_connections 2023-07-20T01:26:56.996 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:26:57.017+0000 7fe59a6c1700 1 -- 172.21.15.183:0/4009085047 wait complete. 2023-07-20T01:26:57.421 DEBUG:teuthology.orchestra.run.smithi183:osd.11> sudo journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.11.service 2023-07-20T01:26:57.424 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2023-07-20T01:26:57.424 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph osd stat -f json 2023-07-20T01:26:57.455 INFO:journalctl@ceph.osd.11.smithi183.stdout:-- Logs begin at Thu 2023-07-20 01:00:08 UTC. -- 2023-07-20T01:26:57.698 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:57 smithi183 ceph-mon[98477]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-20T01:26:57.698 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:57 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:57.698 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:57 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:57.698 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:57 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:26:57.698 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:57 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:26:57.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:57 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:57.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:57 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:57.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:57 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:57.699 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:57 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:26:57.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:57 smithi136 ceph-mon[98083]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-20T01:26:57.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:57 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:57.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:57 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:57.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:57 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:26:57.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:57 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:26:57.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:57 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:57.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:57 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:57.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:57 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:57.898 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:57 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:26:57.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:57 smithi060 ceph-mon[96524]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-20T01:26:57.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:57 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:57.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:57 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:57.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:57 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:26:57.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:57 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:26:57.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:57 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:57.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:57 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:57.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:57 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:26:57.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:57 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:26:57.999 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:26:58.197 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:26:57 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:26:57.994+0000 7f49a334c640 -1 osd.11 0 log_to_monitors true 2023-07-20T01:26:58.759 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:58 smithi183 ceph-mon[98477]: from='osd.11 [v2:172.21.15.183:6824/1157339878,v1:172.21.15.183:6825/1157339878]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T01:26:58.759 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:58 smithi183 ceph-mon[98477]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T01:26:58.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:58 smithi136 ceph-mon[98083]: from='osd.11 [v2:172.21.15.183:6824/1157339878,v1:172.21.15.183:6825/1157339878]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T01:26:58.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:58 smithi136 ceph-mon[98083]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T01:26:58.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:58 smithi060 ceph-mon[96524]: from='osd.11 [v2:172.21.15.183:6824/1157339878,v1:172.21.15.183:6825/1157339878]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T01:26:58.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:58 smithi060 ceph-mon[96524]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-20T01:26:59.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.188+0000 7f352d482700 1 -- 172.21.15.60:0/4081357838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35280fe8e0 msgr2=0x7f35280fed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:26:59.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.188+0000 7f352d482700 1 --2- 172.21.15.60:0/4081357838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35280fe8e0 0x7f35280fed60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3510009990 tx=0x7f3510009ca0 comp rx=0 tx=0).stop 2023-07-20T01:26:59.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.188+0000 7f352d482700 1 -- 172.21.15.60:0/4081357838 shutdown_connections 2023-07-20T01:26:59.165 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.188+0000 7f352d482700 1 --2- 172.21.15.60:0/4081357838 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f35280ff2a0 0x7f3528108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:26:59.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.188+0000 7f352d482700 1 --2- 172.21.15.60:0/4081357838 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3528101e60 0x7f3528102280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:26:59.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.188+0000 7f352d482700 1 --2- 172.21.15.60:0/4081357838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35280fe8e0 0x7f35280fed60 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:26:59.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.188+0000 7f352d482700 1 -- 172.21.15.60:0/4081357838 >> 172.21.15.60:0/4081357838 conn(0x7f35280fa480 msgr2=0x7f35280fc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:26:59.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.188+0000 7f352d482700 1 -- 172.21.15.60:0/4081357838 shutdown_connections 2023-07-20T01:26:59.166 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.188+0000 7f352d482700 1 -- 172.21.15.60:0/4081357838 wait complete. 2023-07-20T01:26:59.171 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.188+0000 7f352d482700 1 Processor -- start 2023-07-20T01:26:59.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.189+0000 7f352d482700 1 -- start start 2023-07-20T01:26:59.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.189+0000 7f352d482700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35280fe8e0 0x7f3528196400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:26:59.172 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.189+0000 7f352d482700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35280ff2a0 0x7f3528196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:26:59.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.189+0000 7f352d482700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3528101e60 0x7f352819d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:26:59.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.189+0000 7f352d482700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f352819df90 con 0x7f35280ff2a0 2023-07-20T01:26:59.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.189+0000 7f352d482700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f352819e0d0 con 0x7f35280fe8e0 2023-07-20T01:26:59.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.189+0000 7f352d482700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f352819e260 con 0x7f3528101e60 2023-07-20T01:26:59.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.189+0000 7f351ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35280ff2a0 0x7f3528196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:26:59.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.189+0000 7f351ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35280ff2a0 0x7f3528196940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42736/0 (socket says 172.21.15.60:42736) 2023-07-20T01:26:59.173 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.189+0000 7f351ffff700 1 -- 172.21.15.60:0/279970994 learned_addr learned my addr 172.21.15.60:0/279970994 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:26:59.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.189+0000 7f3526ffd700 1 --2- 172.21.15.60:0/279970994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35280fe8e0 0x7f3528196400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:26:59.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.189+0000 7f35277fe700 1 --2- 172.21.15.60:0/279970994 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3528101e60 0x7f352819d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:26:59.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.189+0000 7f351ffff700 1 -- 172.21.15.60:0/279970994 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3528101e60 msgr2=0x7f352819d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:26:59.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.189+0000 7f351ffff700 1 --2- 172.21.15.60:0/279970994 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3528101e60 0x7f352819d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:26:59.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.189+0000 7f351ffff700 1 -- 172.21.15.60:0/279970994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35280fe8e0 msgr2=0x7f3528196400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:26:59.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.189+0000 7f351ffff700 1 --2- 172.21.15.60:0/279970994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35280fe8e0 0x7f3528196400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:26:59.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.189+0000 7f351ffff700 1 -- 172.21.15.60:0/279970994 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35100095f0 con 0x7f35280ff2a0 2023-07-20T01:26:59.174 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.190+0000 7f351ffff700 1 --2- 172.21.15.60:0/279970994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35280ff2a0 0x7f3528196940 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3510004730 tx=0x7f3510004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:26:59.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.190+0000 7f3524ff9700 1 -- 172.21.15.60:0/279970994 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351001d070 con 0x7f35280ff2a0 2023-07-20T01:26:59.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.190+0000 7f3524ff9700 1 -- 172.21.15.60:0/279970994 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f3510004c90 con 0x7f35280ff2a0 2023-07-20T01:26:59.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.190+0000 7f352d482700 1 -- 172.21.15.60:0/279970994 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f352819e4e0 con 0x7f35280ff2a0 2023-07-20T01:26:59.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.190+0000 7f3524ff9700 1 -- 172.21.15.60:0/279970994 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3510003a30 con 0x7f35280ff2a0 2023-07-20T01:26:59.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.190+0000 7f352d482700 1 -- 172.21.15.60:0/279970994 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f352819e8f0 con 0x7f35280ff2a0 2023-07-20T01:26:59.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.191+0000 7f352d482700 1 -- 172.21.15.60:0/279970994 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f352810bf20 con 0x7f35280ff2a0 2023-07-20T01:26:59.175 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.192+0000 7f3524ff9700 1 -- 172.21.15.60:0/279970994 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3510005000 con 0x7f35280ff2a0 2023-07-20T01:26:59.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.192+0000 7f3524ff9700 1 --2- 172.21.15.60:0/279970994 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f35140a1150 0x7f35140a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:26:59.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.192+0000 7f3524ff9700 1 -- 172.21.15.60:0/279970994 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6831+0+0 (secure 0 0 0) 0x7f35100cd670 con 0x7f35280ff2a0 2023-07-20T01:26:59.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.192+0000 7f3526ffd700 1 --2- 172.21.15.60:0/279970994 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f35140a1150 0x7f35140a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:26:59.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.193+0000 7f3526ffd700 1 --2- 172.21.15.60:0/279970994 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f35140a1150 0x7f35140a3610 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3528068890 tx=0x7f351800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:26:59.176 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.196+0000 7f3524ff9700 1 -- 172.21.15.60:0/279970994 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3510097420 con 0x7f35280ff2a0 2023-07-20T01:26:59.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.391+0000 7f352d482700 1 -- 172.21.15.60:0/279970994 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f3528107980 con 0x7f35280ff2a0 2023-07-20T01:26:59.369 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.391+0000 7f3524ff9700 1 -- 172.21.15.60:0/279970994 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v75) v1 ==== 74+0+133 (secure 0 0 0) 0x7f351009aa40 con 0x7f35280ff2a0 2023-07-20T01:26:59.369 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:26:59.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.393+0000 7f352d482700 1 -- 172.21.15.60:0/279970994 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f35140a1150 msgr2=0x7f35140a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:26:59.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.393+0000 7f352d482700 1 --2- 172.21.15.60:0/279970994 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f35140a1150 0x7f35140a3610 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3528068890 tx=0x7f351800b410 comp rx=0 tx=0).stop 2023-07-20T01:26:59.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.393+0000 7f352d482700 1 -- 172.21.15.60:0/279970994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35280ff2a0 msgr2=0x7f3528196940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:26:59.371 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.393+0000 7f352d482700 1 --2- 172.21.15.60:0/279970994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35280ff2a0 0x7f3528196940 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3510004730 tx=0x7f3510004760 comp rx=0 tx=0).stop 2023-07-20T01:26:59.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.393+0000 7f352d482700 1 -- 172.21.15.60:0/279970994 shutdown_connections 2023-07-20T01:26:59.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.393+0000 7f352d482700 1 --2- 172.21.15.60:0/279970994 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f35140a1150 0x7f35140a3610 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:26:59.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.393+0000 7f352d482700 1 --2- 172.21.15.60:0/279970994 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3528101e60 0x7f352819d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:26:59.372 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.393+0000 7f352d482700 1 --2- 172.21.15.60:0/279970994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35280fe8e0 0x7f3528196400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:26:59.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.393+0000 7f352d482700 1 --2- 172.21.15.60:0/279970994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35280ff2a0 0x7f3528196940 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:26:59.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.393+0000 7f352d482700 1 -- 172.21.15.60:0/279970994 >> 172.21.15.60:0/279970994 conn(0x7f35280fa480 msgr2=0x7f35280fc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:26:59.373 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.393+0000 7f352d482700 1 -- 172.21.15.60:0/279970994 shutdown_connections 2023-07-20T01:26:59.374 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:26:59.393+0000 7f352d482700 1 -- 172.21.15.60:0/279970994 wait complete. 2023-07-20T01:26:59.674 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:59 smithi060 ceph-mon[96524]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:26:59.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:59 smithi060 ceph-mon[96524]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-20T01:26:59.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:59 smithi060 ceph-mon[96524]: osdmap e75: 12 total, 11 up, 12 in 2023-07-20T01:26:59.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:59 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T01:26:59.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:59 smithi060 ceph-mon[96524]: from='osd.11 [v2:172.21.15.183:6824/1157339878,v1:172.21.15.183:6825/1157339878]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2023-07-20T01:26:59.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:59 smithi060 ceph-mon[96524]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2023-07-20T01:26:59.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:26:59 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/279970994' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T01:26:59.685 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:26:59 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:26:59.531+0000 7f49989c4700 -1 osd.11 0 waiting for initial osdmap 2023-07-20T01:26:59.685 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:26:59 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:26:59.543+0000 7f49947ba700 -1 osd.11 76 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-20T01:26:59.686 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:59 smithi183 ceph-mon[98477]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:26:59.686 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:59 smithi183 ceph-mon[98477]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-20T01:26:59.687 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:59 smithi183 ceph-mon[98477]: osdmap e75: 12 total, 11 up, 12 in 2023-07-20T01:26:59.687 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:59 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T01:26:59.687 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:59 smithi183 ceph-mon[98477]: from='osd.11 [v2:172.21.15.183:6824/1157339878,v1:172.21.15.183:6825/1157339878]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2023-07-20T01:26:59.687 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:59 smithi183 ceph-mon[98477]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2023-07-20T01:26:59.688 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:26:59 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/279970994' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T01:26:59.759 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":75,"num_osds":12,"num_up_osds":11,"osd_up_since":1689816399,"num_in_osds":12,"osd_in_since":1689816403,"num_remapped_pgs":0} 2023-07-20T01:26:59.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:59 smithi136 ceph-mon[98083]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:26:59.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:59 smithi136 ceph-mon[98083]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-20T01:26:59.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:59 smithi136 ceph-mon[98083]: osdmap e75: 12 total, 11 up, 12 in 2023-07-20T01:26:59.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:59 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T01:26:59.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:59 smithi136 ceph-mon[98083]: from='osd.11 [v2:172.21.15.183:6824/1157339878,v1:172.21.15.183:6825/1157339878]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2023-07-20T01:26:59.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:59 smithi136 ceph-mon[98083]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2023-07-20T01:26:59.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:26:59 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/279970994' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T01:27:00.760 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph osd stat -f json 2023-07-20T01:27:00.794 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:00 smithi060 ceph-mon[96524]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2023-07-20T01:27:00.794 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:00 smithi060 ceph-mon[96524]: osdmap e76: 12 total, 11 up, 12 in 2023-07-20T01:27:00.794 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:00 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T01:27:00.794 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:00 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T01:27:00.794 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:00 smithi060 ceph-mon[96524]: Detected new or changed devices on smithi183 2023-07-20T01:27:00.795 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:00 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:00.795 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:00 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:00.795 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:00 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:27:00.795 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:00 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:27:00.795 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:00 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:27:00.795 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:00 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:27:00.795 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:00 smithi060 ceph-mon[96524]: Adjusting osd_memory_target on smithi183 to 4292M 2023-07-20T01:27:00.795 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:00 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:00.796 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:00 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:00.796 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:00 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:27:00.796 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:00 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:00.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:00 smithi136 ceph-mon[98083]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2023-07-20T01:27:00.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:00 smithi136 ceph-mon[98083]: osdmap e76: 12 total, 11 up, 12 in 2023-07-20T01:27:00.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:00 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T01:27:00.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:00 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T01:27:00.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:00 smithi136 ceph-mon[98083]: Detected new or changed devices on smithi183 2023-07-20T01:27:00.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:00 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:00.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:00 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:00.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:00 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:27:00.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:00 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:27:00.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:00 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:27:00.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:00 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:27:00.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:00 smithi136 ceph-mon[98083]: Adjusting osd_memory_target on smithi183 to 4292M 2023-07-20T01:27:00.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:00 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:00.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:00 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:00.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:00 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:27:00.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:00 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:00.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:00 smithi183 ceph-mon[98477]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2023-07-20T01:27:00.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:00 smithi183 ceph-mon[98477]: osdmap e76: 12 total, 11 up, 12 in 2023-07-20T01:27:00.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:00 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T01:27:00.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:00 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T01:27:00.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:00 smithi183 ceph-mon[98477]: Detected new or changed devices on smithi183 2023-07-20T01:27:00.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:00 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:00.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:00 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:00.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:00 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:27:00.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:00 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:27:00.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:00 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:27:00.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:00 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-20T01:27:00.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:00 smithi183 ceph-mon[98477]: Adjusting osd_memory_target on smithi183 to 4292M 2023-07-20T01:27:00.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:00 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:00.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:00 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:00.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:00 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:27:00.951 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:00 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:01.338 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:27:01.839 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:01 smithi060 ceph-mon[96524]: purged_snaps scrub starts 2023-07-20T01:27:01.839 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:01 smithi060 ceph-mon[96524]: purged_snaps scrub ok 2023-07-20T01:27:01.840 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:01 smithi060 ceph-mon[96524]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:27:01.840 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:01 smithi060 ceph-mon[96524]: osd.11 [v2:172.21.15.183:6824/1157339878,v1:172.21.15.183:6825/1157339878] boot 2023-07-20T01:27:01.840 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:01 smithi060 ceph-mon[96524]: osdmap e77: 12 total, 12 up, 12 in 2023-07-20T01:27:01.840 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:01 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T01:27:01.894 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:01 smithi136 ceph-mon[98083]: purged_snaps scrub starts 2023-07-20T01:27:01.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:01 smithi136 ceph-mon[98083]: purged_snaps scrub ok 2023-07-20T01:27:01.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:01 smithi136 ceph-mon[98083]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:27:01.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:01 smithi136 ceph-mon[98083]: osd.11 [v2:172.21.15.183:6824/1157339878,v1:172.21.15.183:6825/1157339878] boot 2023-07-20T01:27:01.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:01 smithi136 ceph-mon[98083]: osdmap e77: 12 total, 12 up, 12 in 2023-07-20T01:27:01.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:01 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T01:27:01.947 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:01 smithi183 ceph-mon[98477]: purged_snaps scrub starts 2023-07-20T01:27:01.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:01 smithi183 ceph-mon[98477]: purged_snaps scrub ok 2023-07-20T01:27:01.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:01 smithi183 ceph-mon[98477]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-20T01:27:01.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:01 smithi183 ceph-mon[98477]: osd.11 [v2:172.21.15.183:6824/1157339878,v1:172.21.15.183:6825/1157339878] boot 2023-07-20T01:27:01.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:01 smithi183 ceph-mon[98477]: osdmap e77: 12 total, 12 up, 12 in 2023-07-20T01:27:01.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:01 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-20T01:27:02.523 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.542+0000 7fea1831e700 1 -- 172.21.15.60:0/509040293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea101012a0 msgr2=0x7fea10107b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:02.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.542+0000 7fea1831e700 1 --2- 172.21.15.60:0/509040293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea101012a0 0x7fea10107b50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fea0800b0d0 tx=0x7fea0800b3e0 comp rx=0 tx=0).stop 2023-07-20T01:27:02.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.542+0000 7fea1831e700 1 -- 172.21.15.60:0/509040293 shutdown_connections 2023-07-20T01:27:02.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.542+0000 7fea1831e700 1 --2- 172.21.15.60:0/509040293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea101012a0 0x7fea10107b50 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:02.524 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.542+0000 7fea1831e700 1 --2- 172.21.15.60:0/509040293 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fea100ff700 0x7fea100ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:02.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.542+0000 7fea1831e700 1 --2- 172.21.15.60:0/509040293 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea10100900 0x7fea10100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:02.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.542+0000 7fea1831e700 1 -- 172.21.15.60:0/509040293 >> 172.21.15.60:0/509040293 conn(0x7fea100fac80 msgr2=0x7fea100fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:02.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.542+0000 7fea1831e700 1 -- 172.21.15.60:0/509040293 shutdown_connections 2023-07-20T01:27:02.525 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.542+0000 7fea1831e700 1 -- 172.21.15.60:0/509040293 wait complete. 2023-07-20T01:27:02.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.543+0000 7fea1831e700 1 Processor -- start 2023-07-20T01:27:02.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.543+0000 7fea1831e700 1 -- start start 2023-07-20T01:27:02.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.543+0000 7fea1831e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea100ff700 0x7fea101953b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:02.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.543+0000 7fea1831e700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fea10100900 0x7fea101958f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:02.526 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.543+0000 7fea1831e700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea101012a0 0x7fea1019c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:02.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.543+0000 7fea1831e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1019cf40 con 0x7fea100ff700 2023-07-20T01:27:02.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.543+0000 7fea1831e700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1019d080 con 0x7fea101012a0 2023-07-20T01:27:02.527 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.543+0000 7fea1831e700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1019d210 con 0x7fea10100900 2023-07-20T01:27:02.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.543+0000 7fea160ba700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea100ff700 0x7fea101953b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:02.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.543+0000 7fea160ba700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea100ff700 0x7fea101953b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42756/0 (socket says 172.21.15.60:42756) 2023-07-20T01:27:02.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.543+0000 7fea160ba700 1 -- 172.21.15.60:0/2247268403 learned_addr learned my addr 172.21.15.60:0/2247268403 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:27:02.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.543+0000 7fea158b9700 1 --2- 172.21.15.60:0/2247268403 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fea10100900 0x7fea101958f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:02.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.543+0000 7fea168bb700 1 --2- 172.21.15.60:0/2247268403 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea101012a0 0x7fea1019c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:02.528 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.544+0000 7fea160ba700 1 -- 172.21.15.60:0/2247268403 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fea10100900 msgr2=0x7fea101958f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:02.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.544+0000 7fea160ba700 1 --2- 172.21.15.60:0/2247268403 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fea10100900 0x7fea101958f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:02.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.544+0000 7fea160ba700 1 -- 172.21.15.60:0/2247268403 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea101012a0 msgr2=0x7fea1019c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:02.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.544+0000 7fea160ba700 1 --2- 172.21.15.60:0/2247268403 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea101012a0 0x7fea1019c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:02.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.544+0000 7fea160ba700 1 -- 172.21.15.60:0/2247268403 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea08009d00 con 0x7fea100ff700 2023-07-20T01:27:02.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.544+0000 7fea168bb700 1 --2- 172.21.15.60:0/2247268403 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea101012a0 0x7fea1019c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T01:27:02.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.544+0000 7fea160ba700 1 --2- 172.21.15.60:0/2247268403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea100ff700 0x7fea101953b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fea0c00b530 tx=0x7fea0c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:02.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.544+0000 7fea037fe700 1 -- 172.21.15.60:0/2247268403 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0c011830 con 0x7fea100ff700 2023-07-20T01:27:02.529 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.544+0000 7fea037fe700 1 -- 172.21.15.60:0/2247268403 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fea0c011e20 con 0x7fea100ff700 2023-07-20T01:27:02.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.544+0000 7fea1831e700 1 -- 172.21.15.60:0/2247268403 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea100683d0 con 0x7fea100ff700 2023-07-20T01:27:02.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.544+0000 7fea037fe700 1 -- 172.21.15.60:0/2247268403 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0c010dd0 con 0x7fea100ff700 2023-07-20T01:27:02.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.544+0000 7fea1831e700 1 -- 172.21.15.60:0/2247268403 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea100689a0 con 0x7fea100ff700 2023-07-20T01:27:02.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.545+0000 7fea1831e700 1 -- 172.21.15.60:0/2247268403 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea10106c70 con 0x7fea100ff700 2023-07-20T01:27:02.530 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.546+0000 7fea037fe700 1 -- 172.21.15.60:0/2247268403 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fea0c011990 con 0x7fea100ff700 2023-07-20T01:27:02.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.547+0000 7fea037fe700 1 --2- 172.21.15.60:0/2247268403 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fe9fc0a11d0 0x7fe9fc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:02.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.547+0000 7fea158b9700 1 --2- 172.21.15.60:0/2247268403 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fe9fc0a11d0 0x7fe9fc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:02.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.547+0000 7fea037fe700 1 -- 172.21.15.60:0/2247268403 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fea0c0cbe50 con 0x7fea100ff700 2023-07-20T01:27:02.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.547+0000 7fea158b9700 1 --2- 172.21.15.60:0/2247268403 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fe9fc0a11d0 0x7fe9fc0a3690 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fea0400ac50 tx=0x7fea0400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:02.531 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.551+0000 7fea037fe700 1 -- 172.21.15.60:0/2247268403 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fea0c095a40 con 0x7fea100ff700 2023-07-20T01:27:02.725 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.748+0000 7fea1831e700 1 -- 172.21.15.60:0/2247268403 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fea1002c970 con 0x7fea100ff700 2023-07-20T01:27:02.726 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.749+0000 7fea037fe700 1 -- 172.21.15.60:0/2247268403 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v78) v1 ==== 74+0+133 (secure 0 0 0) 0x7fea0c099060 con 0x7fea100ff700 2023-07-20T01:27:02.726 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:27:02.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.751+0000 7fea1831e700 1 -- 172.21.15.60:0/2247268403 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fe9fc0a11d0 msgr2=0x7fe9fc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:02.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.751+0000 7fea1831e700 1 --2- 172.21.15.60:0/2247268403 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fe9fc0a11d0 0x7fe9fc0a3690 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fea0400ac50 tx=0x7fea0400a380 comp rx=0 tx=0).stop 2023-07-20T01:27:02.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.751+0000 7fea1831e700 1 -- 172.21.15.60:0/2247268403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea100ff700 msgr2=0x7fea101953b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:02.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.751+0000 7fea1831e700 1 --2- 172.21.15.60:0/2247268403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea100ff700 0x7fea101953b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fea0c00b530 tx=0x7fea0c00b840 comp rx=0 tx=0).stop 2023-07-20T01:27:02.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.751+0000 7fea1831e700 1 -- 172.21.15.60:0/2247268403 shutdown_connections 2023-07-20T01:27:02.728 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.751+0000 7fea1831e700 1 --2- 172.21.15.60:0/2247268403 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fe9fc0a11d0 0x7fe9fc0a3690 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:02.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.751+0000 7fea1831e700 1 --2- 172.21.15.60:0/2247268403 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea101012a0 0x7fea1019c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:02.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.751+0000 7fea1831e700 1 --2- 172.21.15.60:0/2247268403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea100ff700 0x7fea101953b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:02.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.751+0000 7fea1831e700 1 --2- 172.21.15.60:0/2247268403 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fea10100900 0x7fea101958f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:02.730 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.751+0000 7fea1831e700 1 -- 172.21.15.60:0/2247268403 >> 172.21.15.60:0/2247268403 conn(0x7fea100fac80 msgr2=0x7fea10103db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:02.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.751+0000 7fea1831e700 1 -- 172.21.15.60:0/2247268403 shutdown_connections 2023-07-20T01:27:02.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:02.751+0000 7fea1831e700 1 -- 172.21.15.60:0/2247268403 wait complete. 2023-07-20T01:27:03.117 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:02 smithi060 ceph-mon[96524]: osdmap e78: 12 total, 12 up, 12 in 2023-07-20T01:27:03.117 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:02 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/2247268403' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T01:27:03.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:02 smithi136 ceph-mon[98083]: osdmap e78: 12 total, 12 up, 12 in 2023-07-20T01:27:03.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:02 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/2247268403' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T01:27:03.183 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":78,"num_osds":12,"num_up_osds":12,"osd_up_since":1689816420,"num_in_osds":12,"osd_in_since":1689816403,"num_remapped_pgs":0} 2023-07-20T01:27:03.183 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph osd dump --format=json 2023-07-20T01:27:03.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:02 smithi183 ceph-mon[98477]: osdmap e78: 12 total, 12 up, 12 in 2023-07-20T01:27:03.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:02 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/2247268403' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-20T01:27:03.782 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:27:04.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:03 smithi136 ceph-mon[98083]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:04.169 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:03 smithi060 ceph-mon[96524]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:04.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:03 smithi183 ceph-mon[98477]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:04.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.879+0000 7f4a303c9700 1 -- 172.21.15.60:0/3520159592 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a28105ba0 msgr2=0x7f4a2810d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:04.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.879+0000 7f4a303c9700 1 --2- 172.21.15.60:0/3520159592 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a28105ba0 0x7f4a2810d5b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4a200099c0 tx=0x7f4a20009cd0 comp rx=0 tx=0).stop 2023-07-20T01:27:04.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.880+0000 7f4a303c9700 1 -- 172.21.15.60:0/3520159592 shutdown_connections 2023-07-20T01:27:04.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.880+0000 7f4a303c9700 1 --2- 172.21.15.60:0/3520159592 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a28105ba0 0x7f4a2810d5b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:04.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.880+0000 7f4a303c9700 1 --2- 172.21.15.60:0/3520159592 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a28068730 0x7f4a28068b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:04.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.880+0000 7f4a303c9700 1 --2- 172.21.15.60:0/3520159592 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a281051e0 0x7f4a28105660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:04.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.880+0000 7f4a303c9700 1 -- 172.21.15.60:0/3520159592 >> 172.21.15.60:0/3520159592 conn(0x7f4a28076770 msgr2=0x7f4a28076b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:04.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.880+0000 7f4a303c9700 1 -- 172.21.15.60:0/3520159592 shutdown_connections 2023-07-20T01:27:04.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.880+0000 7f4a303c9700 1 -- 172.21.15.60:0/3520159592 wait complete. 2023-07-20T01:27:04.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.880+0000 7f4a303c9700 1 Processor -- start 2023-07-20T01:27:04.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.880+0000 7f4a303c9700 1 -- start start 2023-07-20T01:27:04.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.881+0000 7f4a303c9700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a28068730 0x7f4a2819c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:04.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.881+0000 7f4a303c9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a281051e0 0x7f4a2819ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:04.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.881+0000 7f4a303c9700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a28105ba0 0x7f4a28196620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:04.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.881+0000 7f4a303c9700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a28196c90 con 0x7f4a281051e0 2023-07-20T01:27:04.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.881+0000 7f4a303c9700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a28196e00 con 0x7f4a28105ba0 2023-07-20T01:27:04.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.881+0000 7f4a303c9700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a28196fc0 con 0x7f4a28068730 2023-07-20T01:27:04.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.881+0000 7f4a2d964700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a281051e0 0x7f4a2819ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:04.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.881+0000 7f4a2d964700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a281051e0 0x7f4a2819ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42770/0 (socket says 172.21.15.60:42770) 2023-07-20T01:27:04.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.881+0000 7f4a2d964700 1 -- 172.21.15.60:0/911135964 learned_addr learned my addr 172.21.15.60:0/911135964 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:27:04.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.881+0000 7f4a2e165700 1 --2- 172.21.15.60:0/911135964 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a28068730 0x7f4a2819c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:04.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.881+0000 7f4a2e966700 1 --2- 172.21.15.60:0/911135964 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a28105ba0 0x7f4a28196620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:04.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.881+0000 7f4a2d964700 1 -- 172.21.15.60:0/911135964 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a28068730 msgr2=0x7f4a2819c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:04.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.881+0000 7f4a2d964700 1 --2- 172.21.15.60:0/911135964 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a28068730 0x7f4a2819c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:04.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.881+0000 7f4a2d964700 1 -- 172.21.15.60:0/911135964 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a28105ba0 msgr2=0x7f4a28196620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:04.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.881+0000 7f4a2d964700 1 --2- 172.21.15.60:0/911135964 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a28105ba0 0x7f4a28196620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:04.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.881+0000 7f4a2d964700 1 -- 172.21.15.60:0/911135964 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a200095f0 con 0x7f4a281051e0 2023-07-20T01:27:04.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.881+0000 7f4a2e966700 1 --2- 172.21.15.60:0/911135964 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a28105ba0 0x7f4a28196620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T01:27:04.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.881+0000 7f4a2d964700 1 --2- 172.21.15.60:0/911135964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a281051e0 0x7f4a2819ca70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c00d950 tx=0x7f4a1c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:04.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.881+0000 7f4a277fe700 1 -- 172.21.15.60:0/911135964 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c009940 con 0x7f4a281051e0 2023-07-20T01:27:04.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.881+0000 7f4a277fe700 1 -- 172.21.15.60:0/911135964 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4a1c010460 con 0x7f4a281051e0 2023-07-20T01:27:04.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.881+0000 7f4a303c9700 1 -- 172.21.15.60:0/911135964 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a281972a0 con 0x7f4a281051e0 2023-07-20T01:27:04.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.881+0000 7f4a277fe700 1 -- 172.21.15.60:0/911135964 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c00f620 con 0x7f4a281051e0 2023-07-20T01:27:04.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.881+0000 7f4a303c9700 1 -- 172.21.15.60:0/911135964 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a281a3520 con 0x7f4a281051e0 2023-07-20T01:27:04.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.882+0000 7f4a303c9700 1 -- 172.21.15.60:0/911135964 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a2804e470 con 0x7f4a281051e0 2023-07-20T01:27:04.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.883+0000 7f4a277fe700 1 -- 172.21.15.60:0/911135964 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4a1c0105d0 con 0x7f4a281051e0 2023-07-20T01:27:04.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.884+0000 7f4a277fe700 1 --2- 172.21.15.60:0/911135964 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f4a100a11d0 0x7f4a100a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:04.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.884+0000 7f4a2e165700 1 --2- 172.21.15.60:0/911135964 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f4a100a11d0 0x7f4a100a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:04.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.884+0000 7f4a277fe700 1 -- 172.21.15.60:0/911135964 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4a1c0cc1a0 con 0x7f4a281051e0 2023-07-20T01:27:04.864 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.885+0000 7f4a2e165700 1 --2- 172.21.15.60:0/911135964 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f4a100a11d0 0x7f4a100a3690 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4a18009550 tx=0x7f4a180091f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:04.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:04.888+0000 7f4a277fe700 1 -- 172.21.15.60:0/911135964 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4a1c095d90 con 0x7f4a281051e0 2023-07-20T01:27:05.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:05.084+0000 7f4a303c9700 1 -- 172.21.15.60:0/911135964 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a281a3830 con 0x7f4a281051e0 2023-07-20T01:27:05.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:05.085+0000 7f4a277fe700 1 -- 172.21.15.60:0/911135964 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+18396 (secure 0 0 0) 0x7f4a1c0993b0 con 0x7f4a281051e0 2023-07-20T01:27:05.062 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:27:05.063 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":78,"fsid":"7daf031e-269b-11ee-9b34-001a4aab830c","created":"2023-07-20T01:20:36.943348+0000","modified":"2023-07-20T01:27:01.834100+0000","last_up_change":"2023-07-20T01:27:00.528800+0000","last_in_change":"2023-07-20T01:26:43.393378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T01:23:48.328825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9e5d1e7-1bfe-445f-8907-a1bf68b56c46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":452810841},{"type":"v1","addr":"172.21.15.60:6803","nonce":452810841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":452810841},{"type":"v1","addr":"172.21.15.60:6805","nonce":452810841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":452810841},{"type":"v1","addr":"172.21.15.60:6809","nonce":452810841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":452810841},{"type":"v1","addr":"172.21.15.60:6807","nonce":452810841}]},"public_addr":"172.21.15.60:6803/452810841","cluster_addr":"172.21.15.60:6805/452810841","heartbeat_back_addr":"172.21.15.60:6809/452810841","heartbeat_front_addr":"172.21.15.60:6807/452810841","state":["exists","up"]},{"osd":1,"uuid":"74dc92bd-d5ac-49ed-9918-3a91c53e9486","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":2541801111},{"type":"v1","addr":"172.21.15.136:6801","nonce":2541801111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":2541801111},{"type":"v1","addr":"172.21.15.136:6803","nonce":2541801111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":2541801111},{"type":"v1","addr":"172.21.15.136:6807","nonce":2541801111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":2541801111},{"type":"v1","addr":"172.21.15.136:6805","nonce":2541801111}]},"public_addr":"172.21.15.136:6801/2541801111","cluster_addr":"172.21.15.136:6803/2541801111","heartbeat_back_addr":"172.21.15.136:6807/2541801111","heartbeat_front_addr":"172.21.15.136:6805/2541801111","state":["exists","up"]},{"osd":2,"uuid":"c33a6742-8287-4fb4-b707-7e0d9660fd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":4233496322},{"type":"v1","addr":"172.21.15.183:6801","nonce":4233496322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":4233496322},{"type":"v1","addr":"172.21.15.183:6803","nonce":4233496322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":4233496322},{"type":"v1","addr":"172.21.15.183:6807","nonce":4233496322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":4233496322},{"type":"v1","addr":"172.21.15.183:6805","nonce":4233496322}]},"public_addr":"172.21.15.183:6801/4233496322","cluster_addr":"172.21.15.183:6803/4233496322","heartbeat_back_addr":"172.21.15.183:6807/4233496322","heartbeat_front_addr":"172.21.15.183:6805/4233496322","state":["exists","up"]},{"osd":3,"uuid":"bb99ac8e-19f3-4e76-907f-d1739cbddac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":3285594248},{"type":"v1","addr":"172.21.15.60:6811","nonce":3285594248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3285594248},{"type":"v1","addr":"172.21.15.60:6813","nonce":3285594248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":3285594248},{"type":"v1","addr":"172.21.15.60:6817","nonce":3285594248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3285594248},{"type":"v1","addr":"172.21.15.60:6815","nonce":3285594248}]},"public_addr":"172.21.15.60:6811/3285594248","cluster_addr":"172.21.15.60:6813/3285594248","heartbeat_back_addr":"172.21.15.60:6817/3285594248","heartbeat_front_addr":"172.21.15.60:6815/3285594248","state":["exists","up"]},{"osd":4,"uuid":"5a4e21c8-0151-445f-a94d-852a80b1bce6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2380508636},{"type":"v1","addr":"172.21.15.136:6809","nonce":2380508636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2380508636},{"type":"v1","addr":"172.21.15.136:6811","nonce":2380508636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2380508636},{"type":"v1","addr":"172.21.15.136:6815","nonce":2380508636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2380508636},{"type":"v1","addr":"172.21.15.136:6813","nonce":2380508636}]},"public_addr":"172.21.15.136:6809/2380508636","cluster_addr":"172.21.15.136:6811/2380508636","heartbeat_back_addr":"172.21.15.136:6815/2380508636","heartbeat_front_addr":"172.21.15.136:6813/2380508636","state":["exists","up"]},{"osd":5,"uuid":"a9e51980-2c9e-40ea-ad3f-cb233018455f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2401792837},{"type":"v1","addr":"172.21.15.183:6809","nonce":2401792837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2401792837},{"type":"v1","addr":"172.21.15.183:6811","nonce":2401792837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2401792837},{"type":"v1","addr":"172.21.15.183:6815","nonce":2401792837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2401792837},{"type":"v1","addr":"172.21.15.183:6813","nonce":2401792837}]},"public_addr":"172.21.15.183:6809/2401792837","cluster_addr":"172.21.15.183:6811/2401792837","heartbeat_back_addr":"172.21.15.183:6815/2401792837","heartbeat_front_addr":"172.21.15.183:6813/2401792837","state":["exists","up"]},{"osd":6,"uuid":"58ff5768-d850-40f2-b90c-1098dfdc3256","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":3834224571},{"type":"v1","addr":"172.21.15.60:6819","nonce":3834224571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":3834224571},{"type":"v1","addr":"172.21.15.60:6821","nonce":3834224571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":3834224571},{"type":"v1","addr":"172.21.15.60:6825","nonce":3834224571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":3834224571},{"type":"v1","addr":"172.21.15.60:6823","nonce":3834224571}]},"public_addr":"172.21.15.60:6819/3834224571","cluster_addr":"172.21.15.60:6821/3834224571","heartbeat_back_addr":"172.21.15.60:6825/3834224571","heartbeat_front_addr":"172.21.15.60:6823/3834224571","state":["exists","up"]},{"osd":7,"uuid":"375cd893-5f26-42bb-98ed-30b05911a7af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":404531062},{"type":"v1","addr":"172.21.15.136:6817","nonce":404531062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":404531062},{"type":"v1","addr":"172.21.15.136:6819","nonce":404531062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":404531062},{"type":"v1","addr":"172.21.15.136:6823","nonce":404531062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":404531062},{"type":"v1","addr":"172.21.15.136:6821","nonce":404531062}]},"public_addr":"172.21.15.136:6817/404531062","cluster_addr":"172.21.15.136:6819/404531062","heartbeat_back_addr":"172.21.15.136:6823/404531062","heartbeat_front_addr":"172.21.15.136:6821/404531062","state":["exists","up"]},{"osd":8,"uuid":"68b0051c-0fd3-4ea1-b7df-ea0b8409b8cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":502824503},{"type":"v1","addr":"172.21.15.183:6817","nonce":502824503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":502824503},{"type":"v1","addr":"172.21.15.183:6819","nonce":502824503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":502824503},{"type":"v1","addr":"172.21.15.183:6823","nonce":502824503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":502824503},{"type":"v1","addr":"172.21.15.183:6821","nonce":502824503}]},"public_addr":"172.21.15.183:6817/502824503","cluster_addr":"172.21.15.183:6819/502824503","heartbeat_back_addr":"172.21.15.183:6823/502824503","heartbeat_front_addr":"172.21.15.183:6821/502824503","state":["exists","up"]},{"osd":9,"uuid":"cafd4cc3-258c-4f88-8d3d-0eb4dfa2e2c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":634188302},{"type":"v1","addr":"172.21.15.60:6827","nonce":634188302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":634188302},{"type":"v1","addr":"172.21.15.60:6829","nonce":634188302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":634188302},{"type":"v1","addr":"172.21.15.60:6833","nonce":634188302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":634188302},{"type":"v1","addr":"172.21.15.60:6831","nonce":634188302}]},"public_addr":"172.21.15.60:6827/634188302","cluster_addr":"172.21.15.60:6829/634188302","heartbeat_back_addr":"172.21.15.60:6833/634188302","heartbeat_front_addr":"172.21.15.60:6831/634188302","state":["exists","up"]},{"osd":10,"uuid":"0b092b5a-8512-47d5-928b-995f8dc17353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":247446439},{"type":"v1","addr":"172.21.15.136:6825","nonce":247446439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":247446439},{"type":"v1","addr":"172.21.15.136:6827","nonce":247446439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":247446439},{"type":"v1","addr":"172.21.15.136:6831","nonce":247446439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":247446439},{"type":"v1","addr":"172.21.15.136:6829","nonce":247446439}]},"public_addr":"172.21.15.136:6825/247446439","cluster_addr":"172.21.15.136:6827/247446439","heartbeat_back_addr":"172.21.15.136:6831/247446439","heartbeat_front_addr":"172.21.15.136:6829/247446439","state":["exists","up"]},{"osd":11,"uuid":"8415b1a9-eb18-4b27-ad4f-9a676757bef9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1157339878},{"type":"v1","addr":"172.21.15.183:6825","nonce":1157339878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":1157339878},{"type":"v1","addr":"172.21.15.183:6827","nonce":1157339878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":1157339878},{"type":"v1","addr":"172.21.15.183:6831","nonce":1157339878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1157339878},{"type":"v1","addr":"172.21.15.183:6829","nonce":1157339878}]},"public_addr":"172.21.15.183:6825/1157339878","cluster_addr":"172.21.15.183:6827/1157339878","heartbeat_back_addr":"172.21.15.183:6831/1157339878","heartbeat_front_addr":"172.21.15.183:6829/1157339878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:23:03.185905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:23:24.597431+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:23:45.809074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:24:06.953357+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:24:28.808398+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:24:50.196615+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:25:11.554360+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:25:32.969005+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:25:54.487750+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:26:15.703232+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:26:37.390344+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:26:58.959419+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2364926895":"2023-07-21T01:21:37.803443+0000","172.21.15.60:6800/712953724":"2023-07-21T01:21:37.803443+0000","172.21.15.60:0/3180624926":"2023-07-21T01:21:37.803443+0000","172.21.15.60:6801/712953724":"2023-07-21T01:21:37.803443+0000","172.21.15.60:0/1758646915":"2023-07-21T01:21:11.418252+0000","172.21.15.60:6801/1628989585":"2023-07-21T01:21:11.418252+0000","172.21.15.60:0/884024563":"2023-07-21T01:21:11.418252+0000","172.21.15.60:0/296900825":"2023-07-21T01:21:11.418252+0000","172.21.15.60:6800/1628989585":"2023-07-21T01:21:11.418252+0000","172.21.15.60:0/3556060174":"2023-07-21T01:21:37.803443+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T01:27:05.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:05.087+0000 7f4a303c9700 1 -- 172.21.15.60:0/911135964 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f4a100a11d0 msgr2=0x7f4a100a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:05.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:05.087+0000 7f4a303c9700 1 --2- 172.21.15.60:0/911135964 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f4a100a11d0 0x7f4a100a3690 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4a18009550 tx=0x7f4a180091f0 comp rx=0 tx=0).stop 2023-07-20T01:27:05.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:05.087+0000 7f4a303c9700 1 -- 172.21.15.60:0/911135964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a281051e0 msgr2=0x7f4a2819ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:05.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:05.087+0000 7f4a303c9700 1 --2- 172.21.15.60:0/911135964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a281051e0 0x7f4a2819ca70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c00d950 tx=0x7f4a1c00dc60 comp rx=0 tx=0).stop 2023-07-20T01:27:05.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:05.087+0000 7f4a303c9700 1 -- 172.21.15.60:0/911135964 shutdown_connections 2023-07-20T01:27:05.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:05.087+0000 7f4a303c9700 1 --2- 172.21.15.60:0/911135964 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f4a100a11d0 0x7f4a100a3690 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:05.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:05.087+0000 7f4a303c9700 1 --2- 172.21.15.60:0/911135964 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a28105ba0 0x7f4a28196620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:05.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:05.087+0000 7f4a303c9700 1 --2- 172.21.15.60:0/911135964 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4a28068730 0x7f4a2819c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:05.065 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:05.087+0000 7f4a303c9700 1 --2- 172.21.15.60:0/911135964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a281051e0 0x7f4a2819ca70 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:05.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:05.087+0000 7f4a303c9700 1 -- 172.21.15.60:0/911135964 >> 172.21.15.60:0/911135964 conn(0x7f4a28076770 msgr2=0x7f4a280fd560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:05.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:05.087+0000 7f4a303c9700 1 -- 172.21.15.60:0/911135964 shutdown_connections 2023-07-20T01:27:05.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:05.087+0000 7f4a303c9700 1 -- 172.21.15.60:0/911135964 wait complete. 2023-07-20T01:27:05.531 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-20T01:23:48.328825+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-20T01:27:05.532 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph osd pool get .mgr pg_num 2023-07-20T01:27:06.091 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.a/config 2023-07-20T01:27:06.144 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:05 smithi136 ceph-mon[98083]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:06.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:05 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/911135964' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T01:27:06.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:05 smithi060 ceph-mon[96524]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:06.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:05 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/911135964' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T01:27:06.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:05 smithi183 ceph-mon[98477]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:06.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:05 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/911135964' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T01:27:07.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.443+0000 7f62760ec700 1 -- 172.21.15.60:0/3957717686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6270100a00 msgr2=0x7f6270100e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:07.421 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.443+0000 7f62760ec700 1 --2- 172.21.15.60:0/3957717686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6270100a00 0x7f6270100e80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6258009990 tx=0x7f6258009ca0 comp rx=0 tx=0).stop 2023-07-20T01:27:07.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.443+0000 7f62760ec700 1 -- 172.21.15.60:0/3957717686 shutdown_connections 2023-07-20T01:27:07.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.443+0000 7f62760ec700 1 --2- 172.21.15.60:0/3957717686 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62701013c0 0x7f627010d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:07.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.443+0000 7f62760ec700 1 --2- 172.21.15.60:0/3957717686 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f62701068d0 0x7f6270106cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:07.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.443+0000 7f62760ec700 1 --2- 172.21.15.60:0/3957717686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6270100a00 0x7f6270100e80 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:07.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.443+0000 7f62760ec700 1 -- 172.21.15.60:0/3957717686 >> 172.21.15.60:0/3957717686 conn(0x7f6270078790 msgr2=0x7f6270078ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:07.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.443+0000 7f62760ec700 1 -- 172.21.15.60:0/3957717686 shutdown_connections 2023-07-20T01:27:07.422 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.443+0000 7f62760ec700 1 -- 172.21.15.60:0/3957717686 wait complete. 2023-07-20T01:27:07.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.443+0000 7f62760ec700 1 Processor -- start 2023-07-20T01:27:07.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.444+0000 7f62760ec700 1 -- start start 2023-07-20T01:27:07.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.444+0000 7f62760ec700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6270100a00 0x7f627019c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:07.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.444+0000 7f62760ec700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62701013c0 0x7f627019ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:07.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.444+0000 7f62760ec700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f62701068d0 0x7f6270196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:07.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.444+0000 7f62760ec700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6270196bf0 con 0x7f62701013c0 2023-07-20T01:27:07.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.444+0000 7f62760ec700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6270196d60 con 0x7f6270100a00 2023-07-20T01:27:07.423 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.444+0000 7f62760ec700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6270196f20 con 0x7f62701068d0 2023-07-20T01:27:07.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.444+0000 7f626effd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62701013c0 0x7f627019ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:07.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.444+0000 7f626effd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62701013c0 0x7f627019ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42784/0 (socket says 172.21.15.60:42784) 2023-07-20T01:27:07.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.444+0000 7f626effd700 1 -- 172.21.15.60:0/2142703554 learned_addr learned my addr 172.21.15.60:0/2142703554 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:27:07.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.444+0000 7f626f7fe700 1 --2- 172.21.15.60:0/2142703554 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6270100a00 0x7f627019c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:07.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.444+0000 7f626ffff700 1 --2- 172.21.15.60:0/2142703554 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f62701068d0 0x7f6270196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:07.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.444+0000 7f626effd700 1 -- 172.21.15.60:0/2142703554 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f62701068d0 msgr2=0x7f6270196580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:07.424 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.444+0000 7f626effd700 1 --2- 172.21.15.60:0/2142703554 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f62701068d0 0x7f6270196580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:07.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.444+0000 7f626effd700 1 -- 172.21.15.60:0/2142703554 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6270100a00 msgr2=0x7f627019c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:07.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.444+0000 7f626effd700 1 --2- 172.21.15.60:0/2142703554 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6270100a00 0x7f627019c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:07.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.444+0000 7f626effd700 1 -- 172.21.15.60:0/2142703554 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62580095f0 con 0x7f62701013c0 2023-07-20T01:27:07.425 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.444+0000 7f626effd700 1 --2- 172.21.15.60:0/2142703554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62701013c0 0x7f627019ca40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f62580036c0 tx=0x7f6258005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:07.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.445+0000 7f626cff9700 1 -- 172.21.15.60:0/2142703554 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f625801d070 con 0x7f62701013c0 2023-07-20T01:27:07.426 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.445+0000 7f626cff9700 1 -- 172.21.15.60:0/2142703554 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6258003f40 con 0x7f62701013c0 2023-07-20T01:27:07.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.445+0000 7f626cff9700 1 -- 172.21.15.60:0/2142703554 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6258022c40 con 0x7f62701013c0 2023-07-20T01:27:07.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.445+0000 7f62760ec700 1 -- 172.21.15.60:0/2142703554 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62701971a0 con 0x7f62701013c0 2023-07-20T01:27:07.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.445+0000 7f62760ec700 1 -- 172.21.15.60:0/2142703554 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62701a34f0 con 0x7f62701013c0 2023-07-20T01:27:07.427 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.446+0000 7f62760ec700 1 -- 172.21.15.60:0/2142703554 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f627004e470 con 0x7f62701013c0 2023-07-20T01:27:07.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.447+0000 7f626cff9700 1 -- 172.21.15.60:0/2142703554 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f6258022470 con 0x7f62701013c0 2023-07-20T01:27:07.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.447+0000 7f626cff9700 1 --2- 172.21.15.60:0/2142703554 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f625c0a1150 0x7f625c0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:07.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.447+0000 7f626f7fe700 1 --2- 172.21.15.60:0/2142703554 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f625c0a1150 0x7f625c0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:07.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.447+0000 7f626cff9700 1 -- 172.21.15.60:0/2142703554 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f62580ce060 con 0x7f62701013c0 2023-07-20T01:27:07.428 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.448+0000 7f626f7fe700 1 --2- 172.21.15.60:0/2142703554 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f625c0a1150 0x7f625c0a3610 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6260007be0 tx=0x7f62600073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:07.429 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.451+0000 7f626cff9700 1 -- 172.21.15.60:0/2142703554 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6258097ce0 con 0x7f62701013c0 2023-07-20T01:27:07.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.646+0000 7f62760ec700 1 -- 172.21.15.60:0/2142703554 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6270197a90 con 0x7f62701013c0 2023-07-20T01:27:07.623 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.646+0000 7f626cff9700 1 -- 172.21.15.60:0/2142703554 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7f625809b300 con 0x7f62701013c0 2023-07-20T01:27:07.624 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-07-20T01:27:07.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.648+0000 7f62760ec700 1 -- 172.21.15.60:0/2142703554 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f625c0a1150 msgr2=0x7f625c0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:07.625 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.648+0000 7f62760ec700 1 --2- 172.21.15.60:0/2142703554 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f625c0a1150 0x7f625c0a3610 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6260007be0 tx=0x7f62600073d0 comp rx=0 tx=0).stop 2023-07-20T01:27:07.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.648+0000 7f62760ec700 1 -- 172.21.15.60:0/2142703554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62701013c0 msgr2=0x7f627019ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:07.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.648+0000 7f62760ec700 1 --2- 172.21.15.60:0/2142703554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62701013c0 0x7f627019ca40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f62580036c0 tx=0x7f6258005740 comp rx=0 tx=0).stop 2023-07-20T01:27:07.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.648+0000 7f62760ec700 1 -- 172.21.15.60:0/2142703554 shutdown_connections 2023-07-20T01:27:07.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.648+0000 7f62760ec700 1 --2- 172.21.15.60:0/2142703554 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f625c0a1150 0x7f625c0a3610 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:07.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.648+0000 7f62760ec700 1 --2- 172.21.15.60:0/2142703554 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f62701068d0 0x7f6270196580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:07.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.648+0000 7f62760ec700 1 --2- 172.21.15.60:0/2142703554 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6270100a00 0x7f627019c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:07.626 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.648+0000 7f62760ec700 1 --2- 172.21.15.60:0/2142703554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62701013c0 0x7f627019ca40 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:07.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.648+0000 7f62760ec700 1 -- 172.21.15.60:0/2142703554 >> 172.21.15.60:0/2142703554 conn(0x7f6270078790 msgr2=0x7f62700ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:07.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.648+0000 7f62760ec700 1 -- 172.21.15.60:0/2142703554 shutdown_connections 2023-07-20T01:27:07.627 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:07.648+0000 7f62760ec700 1 -- 172.21.15.60:0/2142703554 wait complete. 2023-07-20T01:27:07.644 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:07 smithi136 ceph-mon[98083]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:07.649 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:07 smithi060 ceph-mon[96524]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:07.701 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:07 smithi183 ceph-mon[98477]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:08.054 INFO:tasks.cephadm:Adding mds.a on smithi060 2023-07-20T01:27:08.054 INFO:tasks.cephadm:Adding mds.d on smithi060 2023-07-20T01:27:08.055 INFO:tasks.cephadm:Adding mds.g on smithi060 2023-07-20T01:27:08.055 INFO:tasks.cephadm:Adding mds.j on smithi060 2023-07-20T01:27:08.055 INFO:tasks.cephadm:Adding mds.b on smithi136 2023-07-20T01:27:08.055 INFO:tasks.cephadm:Adding mds.e on smithi136 2023-07-20T01:27:08.056 INFO:tasks.cephadm:Adding mds.h on smithi136 2023-07-20T01:27:08.056 INFO:tasks.cephadm:Adding mds.k on smithi136 2023-07-20T01:27:08.056 INFO:tasks.cephadm:Adding mds.c on smithi183 2023-07-20T01:27:08.056 INFO:tasks.cephadm:Adding mds.f on smithi183 2023-07-20T01:27:08.056 INFO:tasks.cephadm:Adding mds.i on smithi183 2023-07-20T01:27:08.057 INFO:tasks.cephadm:Adding mds.l on smithi183 2023-07-20T01:27:08.057 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7daf031e-269b-11ee-9b34-001a4aab830c -- ceph orch apply mds all '12;smithi060=a;smithi060=d;smithi060=g;smithi060=j;smithi136=b;smithi136=e;smithi136=h;smithi136=k;smithi183=c;smithi183=f;smithi183=i;smithi183=l' 2023-07-20T01:27:08.627 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/mon.c/config 2023-07-20T01:27:08.644 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:08 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/2142703554' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T01:27:08.674 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:08 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/2142703554' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T01:27:08.697 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:08 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/2142703554' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T01:27:09.447 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:09 smithi183 ceph-mon[98477]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:09.644 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:09 smithi136 ceph-mon[98083]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:09.674 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:09 smithi060 ceph-mon[96524]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:09.819 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.842+0000 7efccfb8a700 1 -- 172.21.15.183:0/1965989301 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efcc81012a0 msgr2=0x7efcc8107b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:09.819 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.842+0000 7efccfb8a700 1 --2- 172.21.15.183:0/1965989301 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efcc81012a0 0x7efcc8107b50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efcc000b0a0 tx=0x7efcc000b3b0 comp rx=0 tx=0).stop 2023-07-20T01:27:09.820 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.842+0000 7efccfb8a700 1 -- 172.21.15.183:0/1965989301 shutdown_connections 2023-07-20T01:27:09.820 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.842+0000 7efccfb8a700 1 --2- 172.21.15.183:0/1965989301 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efcc81012a0 0x7efcc8107b50 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:09.820 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.842+0000 7efccfb8a700 1 --2- 172.21.15.183:0/1965989301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcc80ff700 0x7efcc80ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:09.820 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.842+0000 7efccfb8a700 1 --2- 172.21.15.183:0/1965989301 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efcc8100900 0x7efcc8100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:09.820 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.842+0000 7efccfb8a700 1 -- 172.21.15.183:0/1965989301 >> 172.21.15.183:0/1965989301 conn(0x7efcc80fac80 msgr2=0x7efcc80fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:09.820 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.842+0000 7efccfb8a700 1 -- 172.21.15.183:0/1965989301 shutdown_connections 2023-07-20T01:27:09.821 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.842+0000 7efccfb8a700 1 -- 172.21.15.183:0/1965989301 wait complete. 2023-07-20T01:27:09.822 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.842+0000 7efccfb8a700 1 Processor -- start 2023-07-20T01:27:09.822 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.842+0000 7efccfb8a700 1 -- start start 2023-07-20T01:27:09.822 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.842+0000 7efccfb8a700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efcc80ff700 0x7efcc8068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:09.823 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.842+0000 7efccfb8a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efcc8100900 0x7efcc819b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:09.823 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.842+0000 7efccfb8a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcc81012a0 0x7efcc819d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:09.823 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.842+0000 7efccfb8a700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc819e080 con 0x7efcc81012a0 2023-07-20T01:27:09.823 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.842+0000 7efccfb8a700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc819e1f0 con 0x7efcc8100900 2023-07-20T01:27:09.823 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.842+0000 7efccfb8a700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc819e3b0 con 0x7efcc80ff700 2023-07-20T01:27:09.824 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.843+0000 7efccd926700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efcc80ff700 0x7efcc8068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:09.824 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.843+0000 7efccd926700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efcc80ff700 0x7efcc8068db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:46338/0 (socket says 172.21.15.183:46338) 2023-07-20T01:27:09.824 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.843+0000 7efccd926700 1 -- 172.21.15.183:0/1413130015 learned_addr learned my addr 172.21.15.183:0/1413130015 (peer_addr_for_me v2:172.21.15.183:0/0) 2023-07-20T01:27:09.824 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.843+0000 7efccd125700 1 --2- 172.21.15.183:0/1413130015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efcc8100900 0x7efcc819b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:09.824 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.843+0000 7efcce127700 1 --2- 172.21.15.183:0/1413130015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcc81012a0 0x7efcc819d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:09.824 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.843+0000 7efccd926700 1 -- 172.21.15.183:0/1413130015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efcc8100900 msgr2=0x7efcc819b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:09.825 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.843+0000 7efccd926700 1 --2- 172.21.15.183:0/1413130015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efcc8100900 0x7efcc819b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:09.825 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.843+0000 7efccd926700 1 -- 172.21.15.183:0/1413130015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcc81012a0 msgr2=0x7efcc819d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:09.825 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.843+0000 7efccd926700 1 --2- 172.21.15.183:0/1413130015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcc81012a0 0x7efcc819d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:09.825 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.843+0000 7efccd926700 1 -- 172.21.15.183:0/1413130015 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc0009d00 con 0x7efcc80ff700 2023-07-20T01:27:09.825 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.843+0000 7efccd926700 1 --2- 172.21.15.183:0/1413130015 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efcc80ff700 0x7efcc8068db0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7efcc400b500 tx=0x7efcc400b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:09.825 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.843+0000 7efcbeffd700 1 -- 172.21.15.183:0/1413130015 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc4011830 con 0x7efcc80ff700 2023-07-20T01:27:09.826 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.843+0000 7efccfb8a700 1 -- 172.21.15.183:0/1413130015 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcc819e690 con 0x7efcc80ff700 2023-07-20T01:27:09.826 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.843+0000 7efcbeffd700 1 -- 172.21.15.183:0/1413130015 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7efcc4011e20 con 0x7efcc80ff700 2023-07-20T01:27:09.826 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.843+0000 7efcbeffd700 1 -- 172.21.15.183:0/1413130015 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc4010dd0 con 0x7efcc80ff700 2023-07-20T01:27:09.826 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.843+0000 7efccfb8a700 1 -- 172.21.15.183:0/1413130015 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcc819ec10 con 0x7efcc80ff700 2023-07-20T01:27:09.826 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.844+0000 7efccfb8a700 1 -- 172.21.15.183:0/1413130015 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcc810ae50 con 0x7efcc80ff700 2023-07-20T01:27:09.827 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.845+0000 7efcbeffd700 1 -- 172.21.15.183:0/1413130015 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7efcc4011990 con 0x7efcc80ff700 2023-07-20T01:27:09.827 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.846+0000 7efcbeffd700 1 --2- 172.21.15.183:0/1413130015 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7efcb40a1150 0x7efcb40a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:09.827 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.846+0000 7efcbeffd700 1 -- 172.21.15.183:0/1413130015 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7efcc40cbee0 con 0x7efcc80ff700 2023-07-20T01:27:09.827 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.846+0000 7efccd125700 1 --2- 172.21.15.183:0/1413130015 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7efcb40a1150 0x7efcb40a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:09.828 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.847+0000 7efccd125700 1 --2- 172.21.15.183:0/1413130015 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7efcb40a1150 0x7efcb40a3610 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7efcc8069310 tx=0x7efcb800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:09.828 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:09.850+0000 7efcbeffd700 1 -- 172.21.15.183:0/1413130015 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7efcc4095ad0 con 0x7efcc80ff700 2023-07-20T01:27:10.020 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:10.042+0000 7efccfb8a700 1 -- 172.21.15.183:0/1413130015 --> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi060=a;smithi060=d;smithi060=g;smithi060=j;smithi136=b;smithi136=e;smithi136=h;smithi136=k;smithi183=c;smithi183=f;smithi183=i;smithi183=l", "target": ["mon-mgr", ""]}) v1 -- 0x7efcc8060cb0 con 0x7efcb40a1150 2023-07-20T01:27:10.029 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:10.052+0000 7efcbeffd700 1 -- 172.21.15.183:0/1413130015 <== mgr.14152 v2:172.21.15.60:6800/1961833180 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7efcc8060cb0 con 0x7efcb40a1150 2023-07-20T01:27:10.029 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled mds.all update... 2023-07-20T01:27:10.031 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:10.054+0000 7efccfb8a700 1 -- 172.21.15.183:0/1413130015 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7efcb40a1150 msgr2=0x7efcb40a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:10.031 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:10.054+0000 7efccfb8a700 1 --2- 172.21.15.183:0/1413130015 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7efcb40a1150 0x7efcb40a3610 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7efcc8069310 tx=0x7efcb800a300 comp rx=0 tx=0).stop 2023-07-20T01:27:10.032 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:10.054+0000 7efccfb8a700 1 -- 172.21.15.183:0/1413130015 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efcc80ff700 msgr2=0x7efcc8068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:10.032 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:10.054+0000 7efccfb8a700 1 --2- 172.21.15.183:0/1413130015 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efcc80ff700 0x7efcc8068db0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7efcc400b500 tx=0x7efcc400b810 comp rx=0 tx=0).stop 2023-07-20T01:27:10.032 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:10.054+0000 7efccfb8a700 1 -- 172.21.15.183:0/1413130015 shutdown_connections 2023-07-20T01:27:10.032 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:10.054+0000 7efccfb8a700 1 --2- 172.21.15.183:0/1413130015 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7efcb40a1150 0x7efcb40a3610 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:10.032 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:10.054+0000 7efccfb8a700 1 --2- 172.21.15.183:0/1413130015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcc81012a0 0x7efcc819d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:10.033 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:10.054+0000 7efccfb8a700 1 --2- 172.21.15.183:0/1413130015 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efcc80ff700 0x7efcc8068db0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:10.033 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:10.054+0000 7efccfb8a700 1 --2- 172.21.15.183:0/1413130015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efcc8100900 0x7efcc819b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:10.033 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:10.054+0000 7efccfb8a700 1 -- 172.21.15.183:0/1413130015 >> 172.21.15.183:0/1413130015 conn(0x7efcc80fac80 msgr2=0x7efcc80fceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:10.033 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:10.054+0000 7efccfb8a700 1 -- 172.21.15.183:0/1413130015 shutdown_connections 2023-07-20T01:27:10.033 INFO:teuthology.orchestra.run.smithi183.stderr:2023-07-20T01:27:10.054+0000 7efccfb8a700 1 -- 172.21.15.183:0/1413130015 wait complete. 2023-07-20T01:27:10.472 DEBUG:teuthology.orchestra.run.smithi060:mds.a> sudo journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.a.service 2023-07-20T01:27:10.475 DEBUG:teuthology.orchestra.run.smithi060:mds.d> sudo journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.d.service 2023-07-20T01:27:10.512 INFO:journalctl@ceph.mds.a.smithi060.stdout:-- Logs begin at Thu 2023-07-20 01:00:02 UTC. -- 2023-07-20T01:27:10.513 DEBUG:teuthology.orchestra.run.smithi060:mds.g> sudo journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.g.service 2023-07-20T01:27:10.544 INFO:journalctl@ceph.mds.d.smithi060.stdout:-- Logs begin at Thu 2023-07-20 01:00:02 UTC. -- 2023-07-20T01:27:10.545 DEBUG:teuthology.orchestra.run.smithi060:mds.j> sudo journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.j.service 2023-07-20T01:27:10.574 INFO:journalctl@ceph.mds.g.smithi060.stdout:-- Logs begin at Thu 2023-07-20 01:00:02 UTC. -- 2023-07-20T01:27:10.575 DEBUG:teuthology.orchestra.run.smithi136:mds.b> sudo journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.b.service 2023-07-20T01:27:10.578 DEBUG:teuthology.orchestra.run.smithi136:mds.e> sudo journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.e.service 2023-07-20T01:27:10.602 INFO:journalctl@ceph.mds.j.smithi060.stdout:-- Logs begin at Thu 2023-07-20 01:00:02 UTC. -- 2023-07-20T01:27:10.614 INFO:journalctl@ceph.mds.b.smithi136.stdout:-- Logs begin at Thu 2023-07-20 01:00:24 UTC. -- 2023-07-20T01:27:10.616 DEBUG:teuthology.orchestra.run.smithi136:mds.h> sudo journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.h.service 2023-07-20T01:27:10.645 INFO:journalctl@ceph.mds.e.smithi136.stdout:-- Logs begin at Thu 2023-07-20 01:00:24 UTC. -- 2023-07-20T01:27:10.646 DEBUG:teuthology.orchestra.run.smithi136:mds.k> sudo journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.k.service 2023-07-20T01:27:10.673 INFO:journalctl@ceph.mds.h.smithi136.stdout:-- Logs begin at Thu 2023-07-20 01:00:24 UTC. -- 2023-07-20T01:27:10.674 DEBUG:teuthology.orchestra.run.smithi183:mds.c> sudo journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.c.service 2023-07-20T01:27:10.678 DEBUG:teuthology.orchestra.run.smithi183:mds.f> sudo journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.f.service 2023-07-20T01:27:10.701 INFO:journalctl@ceph.mds.k.smithi136.stdout:-- Logs begin at Thu 2023-07-20 01:00:24 UTC. -- 2023-07-20T01:27:10.708 INFO:journalctl@ceph.mds.c.smithi183.stdout:-- Logs begin at Thu 2023-07-20 01:00:08 UTC. -- 2023-07-20T01:27:10.709 DEBUG:teuthology.orchestra.run.smithi183:mds.i> sudo journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.i.service 2023-07-20T01:27:10.735 INFO:journalctl@ceph.mds.f.smithi183.stdout:-- Logs begin at Thu 2023-07-20 01:00:08 UTC. -- 2023-07-20T01:27:10.736 DEBUG:teuthology.orchestra.run.smithi183:mds.l> sudo journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.l.service 2023-07-20T01:27:10.762 INFO:journalctl@ceph.mds.i.smithi183.stdout:-- Logs begin at Thu 2023-07-20 01:00:08 UTC. -- 2023-07-20T01:27:10.762 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2023-07-20T01:27:10.763 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T01:27:10.790 INFO:journalctl@ceph.mds.l.smithi183.stdout:-- Logs begin at Thu 2023-07-20 01:00:08 UTC. -- 2023-07-20T01:27:10.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.967+0000 7f8462889700 1 -- 172.21.15.60:0/349344341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f845c1011e0 msgr2=0x7f845c107a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:10.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.967+0000 7f8462889700 1 --2- 172.21.15.60:0/349344341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f845c1011e0 0x7f845c107a90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f845400b0d0 tx=0x7f845400b3e0 comp rx=0 tx=0).stop 2023-07-20T01:27:10.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.967+0000 7f8462889700 1 -- 172.21.15.60:0/349344341 shutdown_connections 2023-07-20T01:27:10.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.967+0000 7f8462889700 1 --2- 172.21.15.60:0/349344341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f845c1011e0 0x7f845c107a90 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:10.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.967+0000 7f8462889700 1 --2- 172.21.15.60:0/349344341 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f845c0ff650 0x7f845c0ffa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:10.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.967+0000 7f8462889700 1 --2- 172.21.15.60:0/349344341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f845c100840 0x7f845c100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:10.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.967+0000 7f8462889700 1 -- 172.21.15.60:0/349344341 >> 172.21.15.60:0/349344341 conn(0x7f845c0faba0 msgr2=0x7f845c0fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:10.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.967+0000 7f8462889700 1 -- 172.21.15.60:0/349344341 shutdown_connections 2023-07-20T01:27:10.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.967+0000 7f8462889700 1 -- 172.21.15.60:0/349344341 wait complete. 2023-07-20T01:27:10.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.968+0000 7f8462889700 1 Processor -- start 2023-07-20T01:27:10.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.968+0000 7f8462889700 1 -- start start 2023-07-20T01:27:10.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.968+0000 7f8462889700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f845c0ff650 0x7f845c196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:10.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.968+0000 7f8462889700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f845c100840 0x7f845c196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:10.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.968+0000 7f8462889700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f845c1011e0 0x7f845c19da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:10.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.968+0000 7f8462889700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845c19dfd0 con 0x7f845c0ff650 2023-07-20T01:27:10.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.968+0000 7f8462889700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845c19e110 con 0x7f845c100840 2023-07-20T01:27:10.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.968+0000 7f8462889700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845c19e2a0 con 0x7f845c1011e0 2023-07-20T01:27:10.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.968+0000 7f845bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f845c0ff650 0x7f845c196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:10.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.968+0000 7f845bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f845c0ff650 0x7f845c196450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42796/0 (socket says 172.21.15.60:42796) 2023-07-20T01:27:10.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.968+0000 7f845bfff700 1 -- 172.21.15.60:0/2431966035 learned_addr learned my addr 172.21.15.60:0/2431966035 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:27:10.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.968+0000 7f8460e26700 1 --2- 172.21.15.60:0/2431966035 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f845c1011e0 0x7f845c19da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:10.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.968+0000 7f845b7fe700 1 --2- 172.21.15.60:0/2431966035 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f845c100840 0x7f845c196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:10.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.968+0000 7f845bfff700 1 -- 172.21.15.60:0/2431966035 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f845c1011e0 msgr2=0x7f845c19da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:10.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.968+0000 7f845bfff700 1 --2- 172.21.15.60:0/2431966035 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f845c1011e0 0x7f845c19da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:10.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.968+0000 7f845bfff700 1 -- 172.21.15.60:0/2431966035 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f845c100840 msgr2=0x7f845c196990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:10.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.968+0000 7f845bfff700 1 --2- 172.21.15.60:0/2431966035 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f845c100840 0x7f845c196990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:10.949 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.969+0000 7f845bfff700 1 -- 172.21.15.60:0/2431966035 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8454009d00 con 0x7f845c0ff650 2023-07-20T01:27:10.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.969+0000 7f845bfff700 1 --2- 172.21.15.60:0/2431966035 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f845c0ff650 0x7f845c196450 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f844400d980 tx=0x7f844400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:10.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.969+0000 7f84597fa700 1 -- 172.21.15.60:0/2431966035 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8444009930 con 0x7f845c0ff650 2023-07-20T01:27:10.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.969+0000 7f84597fa700 1 -- 172.21.15.60:0/2431966035 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8444010460 con 0x7f845c0ff650 2023-07-20T01:27:10.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.969+0000 7f8462889700 1 -- 172.21.15.60:0/2431966035 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f845c19e580 con 0x7f845c0ff650 2023-07-20T01:27:10.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.969+0000 7f84597fa700 1 -- 172.21.15.60:0/2431966035 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844400f620 con 0x7f845c0ff650 2023-07-20T01:27:10.950 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.969+0000 7f8462889700 1 -- 172.21.15.60:0/2431966035 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f845c19ea40 con 0x7f845c0ff650 2023-07-20T01:27:10.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.970+0000 7f8462889700 1 -- 172.21.15.60:0/2431966035 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f845c107730 con 0x7f845c0ff650 2023-07-20T01:27:10.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.971+0000 7f84597fa700 1 -- 172.21.15.60:0/2431966035 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8444009a90 con 0x7f845c0ff650 2023-07-20T01:27:10.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.972+0000 7f84597fa700 1 --2- 172.21.15.60:0/2431966035 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f84480a11a0 0x7f84480a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:10.951 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.972+0000 7f845b7fe700 1 --2- 172.21.15.60:0/2431966035 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f84480a11a0 0x7f84480a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:10.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.972+0000 7f84597fa700 1 -- 172.21.15.60:0/2431966035 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f84440cbf40 con 0x7f845c0ff650 2023-07-20T01:27:10.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.972+0000 7f845b7fe700 1 --2- 172.21.15.60:0/2431966035 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f84480a11a0 0x7f84480a3660 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f844c005fd0 tx=0x7f844c005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:10.952 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:10.976+0000 7f84597fa700 1 -- 172.21.15.60:0/2431966035 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8444095b30 con 0x7f845c0ff650 2023-07-20T01:27:11.153 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.177+0000 7f8462889700 1 -- 172.21.15.60:0/2431966035 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f845c02c970 con 0x7f845c0ff650 2023-07-20T01:27:11.154 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.178+0000 7f84597fa700 1 -- 172.21.15.60:0/2431966035 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+18396 (secure 0 0 0) 0x7f8444099150 con 0x7f845c0ff650 2023-07-20T01:27:11.155 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:27:11.155 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":78,"fsid":"7daf031e-269b-11ee-9b34-001a4aab830c","created":"2023-07-20T01:20:36.943348+0000","modified":"2023-07-20T01:27:01.834100+0000","last_up_change":"2023-07-20T01:27:00.528800+0000","last_in_change":"2023-07-20T01:26:43.393378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T01:23:48.328825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9e5d1e7-1bfe-445f-8907-a1bf68b56c46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":452810841},{"type":"v1","addr":"172.21.15.60:6803","nonce":452810841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":452810841},{"type":"v1","addr":"172.21.15.60:6805","nonce":452810841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":452810841},{"type":"v1","addr":"172.21.15.60:6809","nonce":452810841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":452810841},{"type":"v1","addr":"172.21.15.60:6807","nonce":452810841}]},"public_addr":"172.21.15.60:6803/452810841","cluster_addr":"172.21.15.60:6805/452810841","heartbeat_back_addr":"172.21.15.60:6809/452810841","heartbeat_front_addr":"172.21.15.60:6807/452810841","state":["exists","up"]},{"osd":1,"uuid":"74dc92bd-d5ac-49ed-9918-3a91c53e9486","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":2541801111},{"type":"v1","addr":"172.21.15.136:6801","nonce":2541801111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":2541801111},{"type":"v1","addr":"172.21.15.136:6803","nonce":2541801111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":2541801111},{"type":"v1","addr":"172.21.15.136:6807","nonce":2541801111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":2541801111},{"type":"v1","addr":"172.21.15.136:6805","nonce":2541801111}]},"public_addr":"172.21.15.136:6801/2541801111","cluster_addr":"172.21.15.136:6803/2541801111","heartbeat_back_addr":"172.21.15.136:6807/2541801111","heartbeat_front_addr":"172.21.15.136:6805/2541801111","state":["exists","up"]},{"osd":2,"uuid":"c33a6742-8287-4fb4-b707-7e0d9660fd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":4233496322},{"type":"v1","addr":"172.21.15.183:6801","nonce":4233496322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":4233496322},{"type":"v1","addr":"172.21.15.183:6803","nonce":4233496322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":4233496322},{"type":"v1","addr":"172.21.15.183:6807","nonce":4233496322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":4233496322},{"type":"v1","addr":"172.21.15.183:6805","nonce":4233496322}]},"public_addr":"172.21.15.183:6801/4233496322","cluster_addr":"172.21.15.183:6803/4233496322","heartbeat_back_addr":"172.21.15.183:6807/4233496322","heartbeat_front_addr":"172.21.15.183:6805/4233496322","state":["exists","up"]},{"osd":3,"uuid":"bb99ac8e-19f3-4e76-907f-d1739cbddac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":3285594248},{"type":"v1","addr":"172.21.15.60:6811","nonce":3285594248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3285594248},{"type":"v1","addr":"172.21.15.60:6813","nonce":3285594248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":3285594248},{"type":"v1","addr":"172.21.15.60:6817","nonce":3285594248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3285594248},{"type":"v1","addr":"172.21.15.60:6815","nonce":3285594248}]},"public_addr":"172.21.15.60:6811/3285594248","cluster_addr":"172.21.15.60:6813/3285594248","heartbeat_back_addr":"172.21.15.60:6817/3285594248","heartbeat_front_addr":"172.21.15.60:6815/3285594248","state":["exists","up"]},{"osd":4,"uuid":"5a4e21c8-0151-445f-a94d-852a80b1bce6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2380508636},{"type":"v1","addr":"172.21.15.136:6809","nonce":2380508636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2380508636},{"type":"v1","addr":"172.21.15.136:6811","nonce":2380508636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2380508636},{"type":"v1","addr":"172.21.15.136:6815","nonce":2380508636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2380508636},{"type":"v1","addr":"172.21.15.136:6813","nonce":2380508636}]},"public_addr":"172.21.15.136:6809/2380508636","cluster_addr":"172.21.15.136:6811/2380508636","heartbeat_back_addr":"172.21.15.136:6815/2380508636","heartbeat_front_addr":"172.21.15.136:6813/2380508636","state":["exists","up"]},{"osd":5,"uuid":"a9e51980-2c9e-40ea-ad3f-cb233018455f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2401792837},{"type":"v1","addr":"172.21.15.183:6809","nonce":2401792837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2401792837},{"type":"v1","addr":"172.21.15.183:6811","nonce":2401792837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2401792837},{"type":"v1","addr":"172.21.15.183:6815","nonce":2401792837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2401792837},{"type":"v1","addr":"172.21.15.183:6813","nonce":2401792837}]},"public_addr":"172.21.15.183:6809/2401792837","cluster_addr":"172.21.15.183:6811/2401792837","heartbeat_back_addr":"172.21.15.183:6815/2401792837","heartbeat_front_addr":"172.21.15.183:6813/2401792837","state":["exists","up"]},{"osd":6,"uuid":"58ff5768-d850-40f2-b90c-1098dfdc3256","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":3834224571},{"type":"v1","addr":"172.21.15.60:6819","nonce":3834224571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":3834224571},{"type":"v1","addr":"172.21.15.60:6821","nonce":3834224571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":3834224571},{"type":"v1","addr":"172.21.15.60:6825","nonce":3834224571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":3834224571},{"type":"v1","addr":"172.21.15.60:6823","nonce":3834224571}]},"public_addr":"172.21.15.60:6819/3834224571","cluster_addr":"172.21.15.60:6821/3834224571","heartbeat_back_addr":"172.21.15.60:6825/3834224571","heartbeat_front_addr":"172.21.15.60:6823/3834224571","state":["exists","up"]},{"osd":7,"uuid":"375cd893-5f26-42bb-98ed-30b05911a7af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":404531062},{"type":"v1","addr":"172.21.15.136:6817","nonce":404531062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":404531062},{"type":"v1","addr":"172.21.15.136:6819","nonce":404531062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":404531062},{"type":"v1","addr":"172.21.15.136:6823","nonce":404531062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":404531062},{"type":"v1","addr":"172.21.15.136:6821","nonce":404531062}]},"public_addr":"172.21.15.136:6817/404531062","cluster_addr":"172.21.15.136:6819/404531062","heartbeat_back_addr":"172.21.15.136:6823/404531062","heartbeat_front_addr":"172.21.15.136:6821/404531062","state":["exists","up"]},{"osd":8,"uuid":"68b0051c-0fd3-4ea1-b7df-ea0b8409b8cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":502824503},{"type":"v1","addr":"172.21.15.183:6817","nonce":502824503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":502824503},{"type":"v1","addr":"172.21.15.183:6819","nonce":502824503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":502824503},{"type":"v1","addr":"172.21.15.183:6823","nonce":502824503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":502824503},{"type":"v1","addr":"172.21.15.183:6821","nonce":502824503}]},"public_addr":"172.21.15.183:6817/502824503","cluster_addr":"172.21.15.183:6819/502824503","heartbeat_back_addr":"172.21.15.183:6823/502824503","heartbeat_front_addr":"172.21.15.183:6821/502824503","state":["exists","up"]},{"osd":9,"uuid":"cafd4cc3-258c-4f88-8d3d-0eb4dfa2e2c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":634188302},{"type":"v1","addr":"172.21.15.60:6827","nonce":634188302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":634188302},{"type":"v1","addr":"172.21.15.60:6829","nonce":634188302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":634188302},{"type":"v1","addr":"172.21.15.60:6833","nonce":634188302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":634188302},{"type":"v1","addr":"172.21.15.60:6831","nonce":634188302}]},"public_addr":"172.21.15.60:6827/634188302","cluster_addr":"172.21.15.60:6829/634188302","heartbeat_back_addr":"172.21.15.60:6833/634188302","heartbeat_front_addr":"172.21.15.60:6831/634188302","state":["exists","up"]},{"osd":10,"uuid":"0b092b5a-8512-47d5-928b-995f8dc17353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":247446439},{"type":"v1","addr":"172.21.15.136:6825","nonce":247446439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":247446439},{"type":"v1","addr":"172.21.15.136:6827","nonce":247446439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":247446439},{"type":"v1","addr":"172.21.15.136:6831","nonce":247446439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":247446439},{"type":"v1","addr":"172.21.15.136:6829","nonce":247446439}]},"public_addr":"172.21.15.136:6825/247446439","cluster_addr":"172.21.15.136:6827/247446439","heartbeat_back_addr":"172.21.15.136:6831/247446439","heartbeat_front_addr":"172.21.15.136:6829/247446439","state":["exists","up"]},{"osd":11,"uuid":"8415b1a9-eb18-4b27-ad4f-9a676757bef9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1157339878},{"type":"v1","addr":"172.21.15.183:6825","nonce":1157339878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":1157339878},{"type":"v1","addr":"172.21.15.183:6827","nonce":1157339878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":1157339878},{"type":"v1","addr":"172.21.15.183:6831","nonce":1157339878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1157339878},{"type":"v1","addr":"172.21.15.183:6829","nonce":1157339878}]},"public_addr":"172.21.15.183:6825/1157339878","cluster_addr":"172.21.15.183:6827/1157339878","heartbeat_back_addr":"172.21.15.183:6831/1157339878","heartbeat_front_addr":"172.21.15.183:6829/1157339878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:23:03.185905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:23:24.597431+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:23:45.809074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:24:06.953357+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:24:28.808398+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:24:50.196615+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:25:11.554360+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:25:32.969005+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:25:54.487750+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:26:15.703232+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:26:37.390344+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:26:58.959419+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2364926895":"2023-07-21T01:21:37.803443+0000","172.21.15.60:6800/712953724":"2023-07-21T01:21:37.803443+0000","172.21.15.60:0/3180624926":"2023-07-21T01:21:37.803443+0000","172.21.15.60:6801/712953724":"2023-07-21T01:21:37.803443+0000","172.21.15.60:0/1758646915":"2023-07-21T01:21:11.418252+0000","172.21.15.60:6801/1628989585":"2023-07-21T01:21:11.418252+0000","172.21.15.60:0/884024563":"2023-07-21T01:21:11.418252+0000","172.21.15.60:0/296900825":"2023-07-21T01:21:11.418252+0000","172.21.15.60:6800/1628989585":"2023-07-21T01:21:11.418252+0000","172.21.15.60:0/3556060174":"2023-07-21T01:21:37.803443+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T01:27:11.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.180+0000 7f8462889700 1 -- 172.21.15.60:0/2431966035 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f84480a11a0 msgr2=0x7f84480a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:11.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.180+0000 7f8462889700 1 --2- 172.21.15.60:0/2431966035 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f84480a11a0 0x7f84480a3660 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f844c005fd0 tx=0x7f844c005f60 comp rx=0 tx=0).stop 2023-07-20T01:27:11.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.180+0000 7f8462889700 1 -- 172.21.15.60:0/2431966035 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f845c0ff650 msgr2=0x7f845c196450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:11.157 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.180+0000 7f8462889700 1 --2- 172.21.15.60:0/2431966035 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f845c0ff650 0x7f845c196450 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f844400d980 tx=0x7f844400dc90 comp rx=0 tx=0).stop 2023-07-20T01:27:11.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.180+0000 7f8462889700 1 -- 172.21.15.60:0/2431966035 shutdown_connections 2023-07-20T01:27:11.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.180+0000 7f8462889700 1 --2- 172.21.15.60:0/2431966035 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f84480a11a0 0x7f84480a3660 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:11.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.180+0000 7f8462889700 1 --2- 172.21.15.60:0/2431966035 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f845c1011e0 0x7f845c19da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:11.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.180+0000 7f8462889700 1 --2- 172.21.15.60:0/2431966035 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f845c0ff650 0x7f845c196450 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:11.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.180+0000 7f8462889700 1 --2- 172.21.15.60:0/2431966035 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f845c100840 0x7f845c196990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:11.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.180+0000 7f8462889700 1 -- 172.21.15.60:0/2431966035 >> 172.21.15.60:0/2431966035 conn(0x7f845c0faba0 msgr2=0x7f845c0fce00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:11.158 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.180+0000 7f8462889700 1 -- 172.21.15.60:0/2431966035 shutdown_connections 2023-07-20T01:27:11.159 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.180+0000 7f8462889700 1 -- 172.21.15.60:0/2431966035 wait complete. 2023-07-20T01:27:11.169 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:11 smithi060 ceph-mon[96524]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:11.169 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:11 smithi060 ceph-mon[96524]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi060=a;smithi060=d;smithi060=g;smithi060=j;smithi136=b;smithi136=e;smithi136=h;smithi136=k;smithi183=c;smithi183=f;smithi183=i;smithi183=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:27:11.169 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:11 smithi060 ceph-mon[96524]: Saving service mds.all spec with placement smithi060=a;smithi060=d;smithi060=g;smithi060=j;smithi136=b;smithi136=e;smithi136=h;smithi136=k;smithi183=c;smithi183=f;smithi183=i;smithi183=l;count:12 2023-07-20T01:27:11.169 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:11 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:11.169 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:11 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:27:11.170 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:11 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:11.170 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:11 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:27:11.170 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:11 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:11.170 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:11 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:11.170 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:11 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:11.170 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:11 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:11.170 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:11 smithi060 ceph-mon[96524]: Deploying daemon mds.l on smithi183 2023-07-20T01:27:11.172 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-20T01:23:48.328825+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-20T01:27:11.172 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-20T01:27:11.274 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:11 smithi183 ceph-mon[98477]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:11.274 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:11 smithi183 ceph-mon[98477]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi060=a;smithi060=d;smithi060=g;smithi060=j;smithi136=b;smithi136=e;smithi136=h;smithi136=k;smithi183=c;smithi183=f;smithi183=i;smithi183=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:27:11.274 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:11 smithi183 ceph-mon[98477]: Saving service mds.all spec with placement smithi060=a;smithi060=d;smithi060=g;smithi060=j;smithi136=b;smithi136=e;smithi136=h;smithi136=k;smithi183=c;smithi183=f;smithi183=i;smithi183=l;count:12 2023-07-20T01:27:11.275 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:11 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:11.275 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:11 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:27:11.275 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:11 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:11.275 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:11 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:27:11.275 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:11 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:11.275 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:11 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:11.275 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:11 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:11.276 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:11 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:11.276 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:11 smithi183 ceph-mon[98477]: Deploying daemon mds.l on smithi183 2023-07-20T01:27:11.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.323+0000 7efc82242700 1 -- 172.21.15.60:0/2677579701 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc7c0ff660 msgr2=0x7efc7c0ffa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:11.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.323+0000 7efc82242700 1 --2- 172.21.15.60:0/2677579701 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc7c0ff660 0x7efc7c0ffa80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efc700099c0 tx=0x7efc70009cd0 comp rx=0 tx=0).stop 2023-07-20T01:27:11.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.323+0000 7efc82242700 1 -- 172.21.15.60:0/2677579701 shutdown_connections 2023-07-20T01:27:11.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.323+0000 7efc82242700 1 --2- 172.21.15.60:0/2677579701 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc7c1011f0 0x7efc7c107aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:11.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.323+0000 7efc82242700 1 --2- 172.21.15.60:0/2677579701 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc7c0ff660 0x7efc7c0ffa80 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:11.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.323+0000 7efc82242700 1 --2- 172.21.15.60:0/2677579701 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efc7c100850 0x7efc7c100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:11.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.323+0000 7efc82242700 1 -- 172.21.15.60:0/2677579701 >> 172.21.15.60:0/2677579701 conn(0x7efc7c0fabd0 msgr2=0x7efc7c0fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:11.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.323+0000 7efc82242700 1 -- 172.21.15.60:0/2677579701 shutdown_connections 2023-07-20T01:27:11.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.323+0000 7efc82242700 1 -- 172.21.15.60:0/2677579701 wait complete. 2023-07-20T01:27:11.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.324+0000 7efc82242700 1 Processor -- start 2023-07-20T01:27:11.303 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.324+0000 7efc82242700 1 -- start start 2023-07-20T01:27:11.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.325+0000 7efc82242700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc7c0ff660 0x7efc7c196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:11.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.325+0000 7efc82242700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efc7c100850 0x7efc7c1968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:11.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.325+0000 7efc82242700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc7c1011f0 0x7efc7c19d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:11.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.325+0000 7efc82242700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c19df10 con 0x7efc7c0ff660 2023-07-20T01:27:11.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.325+0000 7efc82242700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c19e050 con 0x7efc7c100850 2023-07-20T01:27:11.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.325+0000 7efc82242700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c19e1e0 con 0x7efc7c1011f0 2023-07-20T01:27:11.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.325+0000 7efc7b7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc7c0ff660 0x7efc7c196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:11.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.325+0000 7efc7b7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc7c0ff660 0x7efc7c196390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42810/0 (socket says 172.21.15.60:42810) 2023-07-20T01:27:11.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.325+0000 7efc7b7fe700 1 -- 172.21.15.60:0/2883868464 learned_addr learned my addr 172.21.15.60:0/2883868464 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:27:11.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.325+0000 7efc7bfff700 1 --2- 172.21.15.60:0/2883868464 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc7c1011f0 0x7efc7c19d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:11.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.325+0000 7efc7affd700 1 --2- 172.21.15.60:0/2883868464 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efc7c100850 0x7efc7c1968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:11.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.325+0000 7efc7b7fe700 1 -- 172.21.15.60:0/2883868464 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc7c1011f0 msgr2=0x7efc7c19d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:11.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.325+0000 7efc7b7fe700 1 --2- 172.21.15.60:0/2883868464 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc7c1011f0 0x7efc7c19d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:11.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.325+0000 7efc7b7fe700 1 -- 172.21.15.60:0/2883868464 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efc7c100850 msgr2=0x7efc7c1968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:11.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.325+0000 7efc7b7fe700 1 --2- 172.21.15.60:0/2883868464 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efc7c100850 0x7efc7c1968d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:11.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.325+0000 7efc7b7fe700 1 -- 172.21.15.60:0/2883868464 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc700095f0 con 0x7efc7c0ff660 2023-07-20T01:27:11.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.325+0000 7efc7b7fe700 1 --2- 172.21.15.60:0/2883868464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc7c0ff660 0x7efc7c196390 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7efc70009fd0 tx=0x7efc7000fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:11.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.325+0000 7efc78ff9700 1 -- 172.21.15.60:0/2883868464 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7001d070 con 0x7efc7c0ff660 2023-07-20T01:27:11.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.325+0000 7efc82242700 1 -- 172.21.15.60:0/2883868464 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc7c19e460 con 0x7efc7c0ff660 2023-07-20T01:27:11.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.325+0000 7efc78ff9700 1 -- 172.21.15.60:0/2883868464 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7efc700059a0 con 0x7efc7c0ff660 2023-07-20T01:27:11.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.325+0000 7efc78ff9700 1 -- 172.21.15.60:0/2883868464 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc700173f0 con 0x7efc7c0ff660 2023-07-20T01:27:11.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.326+0000 7efc82242700 1 -- 172.21.15.60:0/2883868464 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc7c19e910 con 0x7efc7c0ff660 2023-07-20T01:27:11.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.326+0000 7efc82242700 1 -- 172.21.15.60:0/2883868464 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc7c1065e0 con 0x7efc7c0ff660 2023-07-20T01:27:11.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.328+0000 7efc78ff9700 1 -- 172.21.15.60:0/2883868464 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7efc70022470 con 0x7efc7c0ff660 2023-07-20T01:27:11.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.328+0000 7efc78ff9700 1 --2- 172.21.15.60:0/2883868464 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7efc640a1150 0x7efc640a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:11.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.328+0000 7efc78ff9700 1 -- 172.21.15.60:0/2883868464 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7efc700cd4f0 con 0x7efc7c0ff660 2023-07-20T01:27:11.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.328+0000 7efc7affd700 1 --2- 172.21.15.60:0/2883868464 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7efc640a1150 0x7efc640a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:11.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.329+0000 7efc7affd700 1 --2- 172.21.15.60:0/2883868464 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7efc640a1150 0x7efc640a3610 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7efc6c00bbd0 tx=0x7efc6c00b5b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:11.311 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.332+0000 7efc78ff9700 1 -- 172.21.15.60:0/2883868464 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7efc7005c780 con 0x7efc7c0ff660 2023-07-20T01:27:11.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:11 smithi136 ceph-mon[98083]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:11.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:11 smithi136 ceph-mon[98083]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi060=a;smithi060=d;smithi060=g;smithi060=j;smithi136=b;smithi136=e;smithi136=h;smithi136=k;smithi183=c;smithi183=f;smithi183=i;smithi183=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-20T01:27:11.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:11 smithi136 ceph-mon[98083]: Saving service mds.all spec with placement smithi060=a;smithi060=d;smithi060=g;smithi060=j;smithi136=b;smithi136=e;smithi136=h;smithi136=k;smithi183=c;smithi183=f;smithi183=i;smithi183=l;count:12 2023-07-20T01:27:11.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:11 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:11.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:11 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:27:11.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:11 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:11.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:11 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:27:11.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:11 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:11.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:11 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:11.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:11 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:11.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:11 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:11.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:11 smithi136 ceph-mon[98083]: Deploying daemon mds.l on smithi183 2023-07-20T01:27:11.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.534+0000 7efc82242700 1 -- 172.21.15.60:0/2883868464 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efc7c02cb00 con 0x7efc7c0ff660 2023-07-20T01:27:11.511 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.534+0000 7efc78ff9700 1 -- 172.21.15.60:0/2883868464 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7efc70097170 con 0x7efc7c0ff660 2023-07-20T01:27:11.511 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-07-20T01:27:11.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.536+0000 7efc82242700 1 -- 172.21.15.60:0/2883868464 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7efc640a1150 msgr2=0x7efc640a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:11.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.536+0000 7efc82242700 1 --2- 172.21.15.60:0/2883868464 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7efc640a1150 0x7efc640a3610 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7efc6c00bbd0 tx=0x7efc6c00b5b0 comp rx=0 tx=0).stop 2023-07-20T01:27:11.513 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.536+0000 7efc82242700 1 -- 172.21.15.60:0/2883868464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc7c0ff660 msgr2=0x7efc7c196390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:11.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.536+0000 7efc82242700 1 --2- 172.21.15.60:0/2883868464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc7c0ff660 0x7efc7c196390 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7efc70009fd0 tx=0x7efc7000fa00 comp rx=0 tx=0).stop 2023-07-20T01:27:11.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.536+0000 7efc82242700 1 -- 172.21.15.60:0/2883868464 shutdown_connections 2023-07-20T01:27:11.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.536+0000 7efc82242700 1 --2- 172.21.15.60:0/2883868464 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7efc640a1150 0x7efc640a3610 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:11.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.536+0000 7efc82242700 1 --2- 172.21.15.60:0/2883868464 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc7c1011f0 0x7efc7c19d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:11.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.536+0000 7efc82242700 1 --2- 172.21.15.60:0/2883868464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc7c0ff660 0x7efc7c196390 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:11.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.536+0000 7efc82242700 1 --2- 172.21.15.60:0/2883868464 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efc7c100850 0x7efc7c1968d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:11.514 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.536+0000 7efc82242700 1 -- 172.21.15.60:0/2883868464 >> 172.21.15.60:0/2883868464 conn(0x7efc7c0fabd0 msgr2=0x7efc7c0fcea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:11.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.536+0000 7efc82242700 1 -- 172.21.15.60:0/2883868464 shutdown_connections 2023-07-20T01:27:11.515 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.536+0000 7efc82242700 1 -- 172.21.15.60:0/2883868464 wait complete. 2023-07-20T01:27:11.526 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T01:27:11.651 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.674+0000 7f988a767700 1 -- 172.21.15.60:0/930221021 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98840ff660 msgr2=0x7f98840ffa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:11.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.674+0000 7f988a767700 1 --2- 172.21.15.60:0/930221021 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98840ff660 0x7f98840ffa80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f98740099c0 tx=0x7f9874009cd0 comp rx=0 tx=0).stop 2023-07-20T01:27:11.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.674+0000 7f988a767700 1 -- 172.21.15.60:0/930221021 shutdown_connections 2023-07-20T01:27:11.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.674+0000 7f988a767700 1 --2- 172.21.15.60:0/930221021 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98841011f0 0x7f9884107aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:11.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.674+0000 7f988a767700 1 --2- 172.21.15.60:0/930221021 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98840ff660 0x7f98840ffa80 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:11.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.674+0000 7f988a767700 1 --2- 172.21.15.60:0/930221021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9884100850 0x7f9884100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:11.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.674+0000 7f988a767700 1 -- 172.21.15.60:0/930221021 >> 172.21.15.60:0/930221021 conn(0x7f98840fab90 msgr2=0x7f98840fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:11.652 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.674+0000 7f988a767700 1 -- 172.21.15.60:0/930221021 shutdown_connections 2023-07-20T01:27:11.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.674+0000 7f988a767700 1 -- 172.21.15.60:0/930221021 wait complete. 2023-07-20T01:27:11.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.675+0000 7f988a767700 1 Processor -- start 2023-07-20T01:27:11.653 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.675+0000 7f988a767700 1 -- start start 2023-07-20T01:27:11.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.675+0000 7f988a767700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98840ff660 0x7f9884195280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:11.654 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.675+0000 7f988a767700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9884100850 0x7f98841957c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:11.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.675+0000 7f988a767700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98841011f0 0x7f988419c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:11.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.675+0000 7f988a767700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988419ce00 con 0x7f98841011f0 2023-07-20T01:27:11.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.675+0000 7f988a767700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988419cf40 con 0x7f98840ff660 2023-07-20T01:27:11.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.675+0000 7f988a767700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988419d0d0 con 0x7f9884100850 2023-07-20T01:27:11.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.675+0000 7f9888d04700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98841011f0 0x7f988419c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:11.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.675+0000 7f9888d04700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98841011f0 0x7f988419c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42828/0 (socket says 172.21.15.60:42828) 2023-07-20T01:27:11.655 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.675+0000 7f9888d04700 1 -- 172.21.15.60:0/3809823101 learned_addr learned my addr 172.21.15.60:0/3809823101 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:27:11.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.675+0000 7f98837fe700 1 --2- 172.21.15.60:0/3809823101 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9884100850 0x7f98841957c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:11.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.675+0000 7f9883fff700 1 --2- 172.21.15.60:0/3809823101 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98840ff660 0x7f9884195280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:11.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.675+0000 7f9888d04700 1 -- 172.21.15.60:0/3809823101 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9884100850 msgr2=0x7f98841957c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:11.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.675+0000 7f9888d04700 1 --2- 172.21.15.60:0/3809823101 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9884100850 0x7f98841957c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:11.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.675+0000 7f9888d04700 1 -- 172.21.15.60:0/3809823101 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98840ff660 msgr2=0x7f9884195280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:11.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.675+0000 7f9888d04700 1 --2- 172.21.15.60:0/3809823101 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98840ff660 0x7f9884195280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:11.656 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.675+0000 7f9888d04700 1 -- 172.21.15.60:0/3809823101 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98740095f0 con 0x7f98841011f0 2023-07-20T01:27:11.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.675+0000 7f9888d04700 1 --2- 172.21.15.60:0/3809823101 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98841011f0 0x7f988419c830 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f987c007ee0 tx=0x7f987c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:11.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.676+0000 7f98817fa700 1 -- 172.21.15.60:0/3809823101 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c017070 con 0x7f98841011f0 2023-07-20T01:27:11.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.676+0000 7f98817fa700 1 -- 172.21.15.60:0/3809823101 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f987c00f040 con 0x7f98841011f0 2023-07-20T01:27:11.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.676+0000 7f988a767700 1 -- 172.21.15.60:0/3809823101 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9884068960 con 0x7f98841011f0 2023-07-20T01:27:11.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.676+0000 7f98817fa700 1 -- 172.21.15.60:0/3809823101 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c005570 con 0x7f98841011f0 2023-07-20T01:27:11.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.676+0000 7f988a767700 1 -- 172.21.15.60:0/3809823101 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9884068e20 con 0x7f98841011f0 2023-07-20T01:27:11.657 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.677+0000 7f988a767700 1 -- 172.21.15.60:0/3809823101 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f988410ad80 con 0x7f98841011f0 2023-07-20T01:27:11.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.678+0000 7f98817fa700 1 -- 172.21.15.60:0/3809823101 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f987c00db20 con 0x7f98841011f0 2023-07-20T01:27:11.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.679+0000 7f98817fa700 1 --2- 172.21.15.60:0/3809823101 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f986c0a33b0 0x7f986c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:11.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.679+0000 7f98817fa700 1 -- 172.21.15.60:0/3809823101 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f987c0cc2e0 con 0x7f98841011f0 2023-07-20T01:27:11.658 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.679+0000 7f9883fff700 1 --2- 172.21.15.60:0/3809823101 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f986c0a33b0 0x7f986c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:11.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.680+0000 7f9883fff700 1 --2- 172.21.15.60:0/3809823101 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f986c0a33b0 0x7f986c0a5870 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9874009990 tx=0x7f987400b9b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:11.659 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.683+0000 7f98817fa700 1 -- 172.21.15.60:0/3809823101 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f987c095e30 con 0x7f98841011f0 2023-07-20T01:27:11.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.928+0000 7f988a767700 1 -- 172.21.15.60:0/3809823101 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9884069460 con 0x7f98841011f0 2023-07-20T01:27:11.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.928+0000 7f98817fa700 1 -- 172.21.15.60:0/3809823101 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7f987c099450 con 0x7f98841011f0 2023-07-20T01:27:11.906 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:27:11.906 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2023-07-20T01:27:11.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.931+0000 7f988a767700 1 -- 172.21.15.60:0/3809823101 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f986c0a33b0 msgr2=0x7f986c0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:11.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.931+0000 7f988a767700 1 --2- 172.21.15.60:0/3809823101 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f986c0a33b0 0x7f986c0a5870 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9874009990 tx=0x7f987400b9b0 comp rx=0 tx=0).stop 2023-07-20T01:27:11.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.931+0000 7f988a767700 1 -- 172.21.15.60:0/3809823101 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98841011f0 msgr2=0x7f988419c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:11.908 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.931+0000 7f988a767700 1 --2- 172.21.15.60:0/3809823101 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98841011f0 0x7f988419c830 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f987c007ee0 tx=0x7f987c00d3b0 comp rx=0 tx=0).stop 2023-07-20T01:27:11.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.931+0000 7f988a767700 1 -- 172.21.15.60:0/3809823101 shutdown_connections 2023-07-20T01:27:11.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.931+0000 7f988a767700 1 --2- 172.21.15.60:0/3809823101 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f986c0a33b0 0x7f986c0a5870 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:11.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.931+0000 7f988a767700 1 --2- 172.21.15.60:0/3809823101 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98841011f0 0x7f988419c830 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:11.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.931+0000 7f988a767700 1 --2- 172.21.15.60:0/3809823101 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98840ff660 0x7f9884195280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:11.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.931+0000 7f988a767700 1 --2- 172.21.15.60:0/3809823101 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9884100850 0x7f98841957c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:11.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.931+0000 7f988a767700 1 -- 172.21.15.60:0/3809823101 >> 172.21.15.60:0/3809823101 conn(0x7f98840fab90 msgr2=0x7f98840fcee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:11.909 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.931+0000 7f988a767700 1 -- 172.21.15.60:0/3809823101 shutdown_connections 2023-07-20T01:27:11.910 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:11.931+0000 7f988a767700 1 -- 172.21.15.60:0/3809823101 wait complete. 2023-07-20T01:27:11.910 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 1 2023-07-20T01:27:11.926 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 20 01:27:11 smithi183 systemd[1]: Starting Ceph mds.l for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:27:12.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:12 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/2431966035' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T01:27:12.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:12 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/2883868464' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T01:27:12.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:12 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/3809823101' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:12.179 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 20 01:27:11 smithi183 podman[123480]: 2023-07-20T01:27:12.179 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:12 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/2431966035' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T01:27:12.180 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:12 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/2883868464' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T01:27:12.180 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:12 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/3809823101' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:12.394 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:12 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/2431966035' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T01:27:12.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:12 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/2883868464' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T01:27:12.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:12 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/3809823101' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:12.447 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 20 01:27:12 smithi183 ceph-mds[123520]: -- 172.21.15.183:0/4264738480 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c7248011e0 con 0x55c7247d6800 2023-07-20T01:27:12.448 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 20 01:27:12 smithi183 bash[123480]: 5a6c2ed2ae6bc2dbf93278a9c7d0fb8e5c810edc5f8b92e5947a629a03576ac6 2023-07-20T01:27:12.448 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 20 01:27:12 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-l[123496]: starting mds.l at 2023-07-20T01:27:12.448 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 20 01:27:12 smithi183 systemd[1]: Started Ceph mds.l for 7daf031e-269b-11ee-9b34-001a4aab830c. 2023-07-20T01:27:13.447 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:13 smithi183 ceph-mon[98477]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:13.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:13 smithi183 ceph-mon[98477]: mds.? [v2:172.21.15.183:6832/3759595107,v1:172.21.15.183:6833/3759595107] up:boot 2023-07-20T01:27:13.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:13 smithi183 ceph-mon[98477]: fsmap 1 up:standby 2023-07-20T01:27:13.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:13 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-20T01:27:13.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:13 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:13.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:13 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:13.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:13 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:13.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:13 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:13.449 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:13 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:13.449 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:13 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:13.449 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:13 smithi183 ceph-mon[98477]: Deploying daemon mds.k on smithi136 2023-07-20T01:27:13.524 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:13 smithi136 ceph-mon[98083]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:13.524 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:13 smithi136 ceph-mon[98083]: mds.? [v2:172.21.15.183:6832/3759595107,v1:172.21.15.183:6833/3759595107] up:boot 2023-07-20T01:27:13.524 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:13 smithi136 ceph-mon[98083]: fsmap 1 up:standby 2023-07-20T01:27:13.525 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:13 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-20T01:27:13.525 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:13 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:13.525 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:13 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:13.525 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:13 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:13.525 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:13 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:13.526 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:13 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:13.526 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:13 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:13.526 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:13 smithi136 ceph-mon[98083]: Deploying daemon mds.k on smithi136 2023-07-20T01:27:13.674 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:13 smithi060 ceph-mon[96524]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:13.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:13 smithi060 ceph-mon[96524]: mds.? [v2:172.21.15.183:6832/3759595107,v1:172.21.15.183:6833/3759595107] up:boot 2023-07-20T01:27:13.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:13 smithi060 ceph-mon[96524]: fsmap 1 up:standby 2023-07-20T01:27:13.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:13 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-20T01:27:13.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:13 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:13.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:13 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:13.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:13 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:13.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:13 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:13.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:13 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:13.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:13 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:13.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:13 smithi060 ceph-mon[96524]: Deploying daemon mds.k on smithi136 2023-07-20T01:27:13.895 INFO:journalctl@ceph.mds.k.smithi136.stdout:Jul 20 01:27:13 smithi136 systemd[1]: Starting Ceph mds.k for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:27:13.921 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T01:27:14.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.084+0000 7f8fa59bb700 1 -- 172.21.15.60:0/1746299556 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8fa01064e0 msgr2=0x7f8fa01088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:14.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.084+0000 7f8fa59bb700 1 --2- 172.21.15.60:0/1746299556 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8fa01064e0 0x7f8fa01088f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8f9800b0a0 tx=0x7f8f9800b3b0 comp rx=0 tx=0).stop 2023-07-20T01:27:14.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.084+0000 7f8fa59bb700 1 -- 172.21.15.60:0/1746299556 shutdown_connections 2023-07-20T01:27:14.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.084+0000 7f8fa59bb700 1 --2- 172.21.15.60:0/1746299556 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8fa01064e0 0x7f8fa01088f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:14.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.084+0000 7f8fa59bb700 1 --2- 172.21.15.60:0/1746299556 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8fa0101280 0x7f8fa0103670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:14.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.084+0000 7f8fa59bb700 1 --2- 172.21.15.60:0/1746299556 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fa0103bb0 0x7f8fa0105fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:14.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.084+0000 7f8fa59bb700 1 -- 172.21.15.60:0/1746299556 >> 172.21.15.60:0/1746299556 conn(0x7f8fa00fabf0 msgr2=0x7f8fa00fd050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:14.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.084+0000 7f8fa59bb700 1 -- 172.21.15.60:0/1746299556 shutdown_connections 2023-07-20T01:27:14.064 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.084+0000 7f8fa59bb700 1 -- 172.21.15.60:0/1746299556 wait complete. 2023-07-20T01:27:14.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.085+0000 7f8fa59bb700 1 Processor -- start 2023-07-20T01:27:14.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.085+0000 7f8fa59bb700 1 -- start start 2023-07-20T01:27:14.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.085+0000 7f8fa59bb700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fa0101280 0x7f8fa0196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:14.066 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.085+0000 7f8fa59bb700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8fa0103bb0 0x7f8fa0196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:14.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.085+0000 7f8fa59bb700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8fa01064e0 0x7f8fa019da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:14.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.085+0000 7f8fa59bb700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa019dfd0 con 0x7f8fa0103bb0 2023-07-20T01:27:14.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.085+0000 7f8fa59bb700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa019e110 con 0x7f8fa01064e0 2023-07-20T01:27:14.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.085+0000 7f8fa59bb700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa019e2a0 con 0x7f8fa0101280 2023-07-20T01:27:14.068 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.085+0000 7f8f9e7fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8fa0103bb0 0x7f8fa0196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:14.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.085+0000 7f8f9e7fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8fa0103bb0 0x7f8fa0196990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42840/0 (socket says 172.21.15.60:42840) 2023-07-20T01:27:14.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.085+0000 7f8f9e7fc700 1 -- 172.21.15.60:0/2323467711 learned_addr learned my addr 172.21.15.60:0/2323467711 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:27:14.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.086+0000 7f8f9effd700 1 --2- 172.21.15.60:0/2323467711 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fa0101280 0x7f8fa0196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:14.069 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.086+0000 7f8f9f7fe700 1 --2- 172.21.15.60:0/2323467711 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8fa01064e0 0x7f8fa019da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:14.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.086+0000 7f8f9e7fc700 1 -- 172.21.15.60:0/2323467711 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fa0101280 msgr2=0x7f8fa0196450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:14.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.086+0000 7f8f9e7fc700 1 --2- 172.21.15.60:0/2323467711 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fa0101280 0x7f8fa0196450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:14.070 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.086+0000 7f8f9e7fc700 1 -- 172.21.15.60:0/2323467711 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8fa01064e0 msgr2=0x7f8fa019da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:14.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.086+0000 7f8f9e7fc700 1 --2- 172.21.15.60:0/2323467711 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8fa01064e0 0x7f8fa019da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:14.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.086+0000 7f8f9e7fc700 1 -- 172.21.15.60:0/2323467711 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f98009d00 con 0x7f8fa0103bb0 2023-07-20T01:27:14.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.086+0000 7f8f9f7fe700 1 --2- 172.21.15.60:0/2323467711 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8fa01064e0 0x7f8fa019da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T01:27:14.071 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.086+0000 7f8f9e7fc700 1 --2- 172.21.15.60:0/2323467711 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8fa0103bb0 0x7f8fa0196990 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8f8800d950 tx=0x7f8f8800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:14.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.086+0000 7f8fa49b9700 1 -- 172.21.15.60:0/2323467711 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f88009940 con 0x7f8fa0103bb0 2023-07-20T01:27:14.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.086+0000 7f8fa49b9700 1 -- 172.21.15.60:0/2323467711 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8f88010460 con 0x7f8fa0103bb0 2023-07-20T01:27:14.072 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.086+0000 7f8fa49b9700 1 -- 172.21.15.60:0/2323467711 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8800f620 con 0x7f8fa0103bb0 2023-07-20T01:27:14.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.086+0000 7f8fa59bb700 1 -- 172.21.15.60:0/2323467711 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fa019e580 con 0x7f8fa0103bb0 2023-07-20T01:27:14.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.086+0000 7f8fa59bb700 1 -- 172.21.15.60:0/2323467711 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fa019ea90 con 0x7f8fa0103bb0 2023-07-20T01:27:14.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.087+0000 7f8fa59bb700 1 -- 172.21.15.60:0/2323467711 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fa00fc770 con 0x7f8fa0103bb0 2023-07-20T01:27:14.073 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.088+0000 7f8fa49b9700 1 -- 172.21.15.60:0/2323467711 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8f880105d0 con 0x7f8fa0103bb0 2023-07-20T01:27:14.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.089+0000 7f8fa49b9700 1 --2- 172.21.15.60:0/2323467711 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f8f8c0a1220 0x7f8f8c0a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:14.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.089+0000 7f8f9effd700 1 --2- 172.21.15.60:0/2323467711 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f8f8c0a1220 0x7f8f8c0a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:14.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.089+0000 7f8fa49b9700 1 -- 172.21.15.60:0/2323467711 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8f880cc2e0 con 0x7f8fa0103bb0 2023-07-20T01:27:14.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.089+0000 7f8f9effd700 1 --2- 172.21.15.60:0/2323467711 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f8f8c0a1220 0x7f8f8c0a36e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8f90009c60 tx=0x7f8f90009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:14.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.093+0000 7f8fa49b9700 1 -- 172.21.15.60:0/2323467711 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8f88095e30 con 0x7f8fa0103bb0 2023-07-20T01:27:14.313 INFO:journalctl@ceph.mds.k.smithi136.stdout:Jul 20 01:27:14 smithi136 podman[122126]: 2023-07-20T01:27:14.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.339+0000 7f8fa59bb700 1 -- 172.21.15.60:0/2323467711 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fa002c970 con 0x7f8fa0103bb0 2023-07-20T01:27:14.317 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.339+0000 7f8fa49b9700 1 -- 172.21.15.60:0/2323467711 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1260 (secure 0 0 0) 0x7f8f88099450 con 0x7f8fa0103bb0 2023-07-20T01:27:14.317 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:27:14.318 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/3759595107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3759595107},{"type":"v1","addr":"172.21.15.183:6833","nonce":3759595107}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-07-20T01:27:14.319 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.342+0000 7f8fa59bb700 1 -- 172.21.15.60:0/2323467711 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f8f8c0a1220 msgr2=0x7f8f8c0a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:14.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.342+0000 7f8fa59bb700 1 --2- 172.21.15.60:0/2323467711 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f8f8c0a1220 0x7f8f8c0a36e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8f90009c60 tx=0x7f8f90009260 comp rx=0 tx=0).stop 2023-07-20T01:27:14.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.342+0000 7f8fa59bb700 1 -- 172.21.15.60:0/2323467711 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8fa0103bb0 msgr2=0x7f8fa0196990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:14.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.342+0000 7f8fa59bb700 1 --2- 172.21.15.60:0/2323467711 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8fa0103bb0 0x7f8fa0196990 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8f8800d950 tx=0x7f8f8800dc60 comp rx=0 tx=0).stop 2023-07-20T01:27:14.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.342+0000 7f8fa59bb700 1 -- 172.21.15.60:0/2323467711 shutdown_connections 2023-07-20T01:27:14.320 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.342+0000 7f8fa59bb700 1 --2- 172.21.15.60:0/2323467711 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f8f8c0a1220 0x7f8f8c0a36e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:14.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.342+0000 7f8fa59bb700 1 --2- 172.21.15.60:0/2323467711 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8fa01064e0 0x7f8fa019da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:14.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.342+0000 7f8fa59bb700 1 --2- 172.21.15.60:0/2323467711 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fa0101280 0x7f8fa0196450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:14.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.342+0000 7f8fa59bb700 1 --2- 172.21.15.60:0/2323467711 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8fa0103bb0 0x7f8fa0196990 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:14.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.342+0000 7f8fa59bb700 1 -- 172.21.15.60:0/2323467711 >> 172.21.15.60:0/2323467711 conn(0x7f8fa00fabf0 msgr2=0x7f8fa0101ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:14.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.342+0000 7f8fa59bb700 1 -- 172.21.15.60:0/2323467711 shutdown_connections 2023-07-20T01:27:14.321 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:14.342+0000 7f8fa59bb700 1 -- 172.21.15.60:0/2323467711 wait complete. 2023-07-20T01:27:14.322 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 2 2023-07-20T01:27:14.644 INFO:journalctl@ceph.mds.k.smithi136.stdout:Jul 20 01:27:14 smithi136 ceph-mds[122166]: -- 172.21.15.136:0/2671094472 <== mon.2 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563d5749b1e0 con 0x563d57471000 2023-07-20T01:27:14.644 INFO:journalctl@ceph.mds.k.smithi136.stdout:Jul 20 01:27:14 smithi136 bash[122126]: 15a519141f22ac5e2bd571404e5f342ce6393f4838ea0726b32c80b864eca230 2023-07-20T01:27:14.645 INFO:journalctl@ceph.mds.k.smithi136.stdout:Jul 20 01:27:14 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-k[122142]: starting mds.k at 2023-07-20T01:27:14.645 INFO:journalctl@ceph.mds.k.smithi136.stdout:Jul 20 01:27:14 smithi136 systemd[1]: Started Ceph mds.k for 7daf031e-269b-11ee-9b34-001a4aab830c. 2023-07-20T01:27:15.363 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:15 smithi060 ceph-mon[96524]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:15.363 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:15 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/2323467711' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:15.364 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:15 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:15.364 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:15 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:15.364 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:15 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:15.364 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:15 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:15.364 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:15 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:15.364 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:15 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:15.364 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:15 smithi060 ceph-mon[96524]: Deploying daemon mds.j on smithi060 2023-07-20T01:27:15.447 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:15 smithi183 ceph-mon[98477]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:15.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:15 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/2323467711' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:15.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:15 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:15.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:15 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:15.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:15 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:15.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:15 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:15.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:15 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:15.449 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:15 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:15.449 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:15 smithi183 ceph-mon[98477]: Deploying daemon mds.j on smithi060 2023-07-20T01:27:15.644 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:15 smithi136 ceph-mon[98083]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:15.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:15 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/2323467711' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:15.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:15 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:15.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:15 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:15.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:15 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:15.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:15 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:15.646 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:15 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:15.646 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:15 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:15.646 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:15 smithi136 ceph-mon[98083]: Deploying daemon mds.j on smithi060 2023-07-20T01:27:16.174 INFO:journalctl@ceph.mds.j.smithi060.stdout:Jul 20 01:27:15 smithi060 systemd[1]: Starting Ceph mds.j for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:27:16.333 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T01:27:16.440 INFO:journalctl@ceph.mds.j.smithi060.stdout:Jul 20 01:27:16 smithi060 podman[129719]: 2023-07-20T01:27:16.440 INFO:journalctl@ceph.mds.j.smithi060.stdout:Jul 20 01:27:16 smithi060 ceph-mds[129782]: -- 172.21.15.60:0/2404548150 <== mon.0 v2:172.21.15.60:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5564032cf1e0 con 0x5564032a4800 2023-07-20T01:27:16.441 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:16 smithi060 ceph-mon[96524]: mds.? [v2:172.21.15.136:6832/1706932385,v1:172.21.15.136:6833/1706932385] up:boot 2023-07-20T01:27:16.441 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:16 smithi060 ceph-mon[96524]: fsmap 2 up:standby 2023-07-20T01:27:16.441 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:16 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-20T01:27:16.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.491+0000 7fb191f65700 1 -- 172.21.15.60:0/191943367 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18c100850 msgr2=0x7fb18c100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:16.471 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.491+0000 7fb191f65700 1 --2- 172.21.15.60:0/191943367 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18c100850 0x7fb18c100cb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb1800099c0 tx=0x7fb180009cd0 comp rx=0 tx=0).stop 2023-07-20T01:27:16.472 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.491+0000 7fb191f65700 1 -- 172.21.15.60:0/191943367 shutdown_connections 2023-07-20T01:27:16.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.491+0000 7fb191f65700 1 --2- 172.21.15.60:0/191943367 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb18c1011f0 0x7fb18c107aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:16.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.491+0000 7fb191f65700 1 --2- 172.21.15.60:0/191943367 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb18c0ff660 0x7fb18c0ffa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:16.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.491+0000 7fb191f65700 1 --2- 172.21.15.60:0/191943367 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18c100850 0x7fb18c100cb0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:16.473 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.491+0000 7fb191f65700 1 -- 172.21.15.60:0/191943367 >> 172.21.15.60:0/191943367 conn(0x7fb18c0fab90 msgr2=0x7fb18c0fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:16.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.491+0000 7fb191f65700 1 -- 172.21.15.60:0/191943367 shutdown_connections 2023-07-20T01:27:16.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.491+0000 7fb191f65700 1 -- 172.21.15.60:0/191943367 wait complete. 2023-07-20T01:27:16.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.492+0000 7fb191f65700 1 Processor -- start 2023-07-20T01:27:16.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.492+0000 7fb191f65700 1 -- start start 2023-07-20T01:27:16.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.492+0000 7fb191f65700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb18c0ff660 0x7fb18c196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:16.474 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.492+0000 7fb191f65700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18c100850 0x7fb18c1968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:16.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.492+0000 7fb191f65700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb18c1011f0 0x7fb18c19d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:16.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.492+0000 7fb191f65700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb18c19def0 con 0x7fb18c100850 2023-07-20T01:27:16.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.492+0000 7fb191f65700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb18c19e030 con 0x7fb18c0ff660 2023-07-20T01:27:16.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.492+0000 7fb191f65700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb18c19e1c0 con 0x7fb18c1011f0 2023-07-20T01:27:16.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.492+0000 7fb18affd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18c100850 0x7fb18c1968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:16.475 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.492+0000 7fb18affd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18c100850 0x7fb18c1968b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42866/0 (socket says 172.21.15.60:42866) 2023-07-20T01:27:16.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.492+0000 7fb18affd700 1 -- 172.21.15.60:0/1808965751 learned_addr learned my addr 172.21.15.60:0/1808965751 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:27:16.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.492+0000 7fb18bfff700 1 --2- 172.21.15.60:0/1808965751 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb18c1011f0 0x7fb18c19d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:16.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.492+0000 7fb18b7fe700 1 --2- 172.21.15.60:0/1808965751 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb18c0ff660 0x7fb18c196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:16.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.492+0000 7fb18affd700 1 -- 172.21.15.60:0/1808965751 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb18c1011f0 msgr2=0x7fb18c19d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:16.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.493+0000 7fb18affd700 1 --2- 172.21.15.60:0/1808965751 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb18c1011f0 0x7fb18c19d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:16.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.493+0000 7fb18affd700 1 -- 172.21.15.60:0/1808965751 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb18c0ff660 msgr2=0x7fb18c196370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:16.476 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.493+0000 7fb18affd700 1 --2- 172.21.15.60:0/1808965751 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb18c0ff660 0x7fb18c196370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:16.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.493+0000 7fb18affd700 1 -- 172.21.15.60:0/1808965751 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1800095f0 con 0x7fb18c100850 2023-07-20T01:27:16.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.493+0000 7fb18affd700 1 --2- 172.21.15.60:0/1808965751 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18c100850 0x7fb18c1968b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb180009fd0 tx=0x7fb18000fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:16.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.493+0000 7fb188ff9700 1 -- 172.21.15.60:0/1808965751 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18001d070 con 0x7fb18c100850 2023-07-20T01:27:16.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.493+0000 7fb188ff9700 1 -- 172.21.15.60:0/1808965751 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb1800059a0 con 0x7fb18c100850 2023-07-20T01:27:16.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.493+0000 7fb191f65700 1 -- 172.21.15.60:0/1808965751 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb18c19e440 con 0x7fb18c100850 2023-07-20T01:27:16.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.493+0000 7fb188ff9700 1 -- 172.21.15.60:0/1808965751 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1800173f0 con 0x7fb18c100850 2023-07-20T01:27:16.477 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.493+0000 7fb191f65700 1 -- 172.21.15.60:0/1808965751 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb18c19e940 con 0x7fb18c100850 2023-07-20T01:27:16.478 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.494+0000 7fb191f65700 1 -- 172.21.15.60:0/1808965751 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb18c10ad20 con 0x7fb18c100850 2023-07-20T01:27:16.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.495+0000 7fb188ff9700 1 -- 172.21.15.60:0/1808965751 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb180022470 con 0x7fb18c100850 2023-07-20T01:27:16.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.496+0000 7fb188ff9700 1 --2- 172.21.15.60:0/1808965751 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fb1740a1220 0x7fb1740a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:16.479 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.496+0000 7fb18b7fe700 1 --2- 172.21.15.60:0/1808965751 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fb1740a1220 0x7fb1740a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:16.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.496+0000 7fb188ff9700 1 -- 172.21.15.60:0/1808965751 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb1800cd580 con 0x7fb18c100850 2023-07-20T01:27:16.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.496+0000 7fb18b7fe700 1 --2- 172.21.15.60:0/1808965751 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fb1740a1220 0x7fb1740a36e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb18c1006b0 tx=0x7fb184009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:16.480 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.500+0000 7fb188ff9700 1 -- 172.21.15.60:0/1808965751 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb180097200 con 0x7fb18c100850 2023-07-20T01:27:16.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:16 smithi136 ceph-mon[98083]: mds.? [v2:172.21.15.136:6832/1706932385,v1:172.21.15.136:6833/1706932385] up:boot 2023-07-20T01:27:16.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:16 smithi136 ceph-mon[98083]: fsmap 2 up:standby 2023-07-20T01:27:16.645 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:16 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-20T01:27:16.697 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:16 smithi183 ceph-mon[98477]: mds.? [v2:172.21.15.136:6832/1706932385,v1:172.21.15.136:6833/1706932385] up:boot 2023-07-20T01:27:16.698 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:16 smithi183 ceph-mon[98477]: fsmap 2 up:standby 2023-07-20T01:27:16.698 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:16 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-20T01:27:16.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.751+0000 7fb191f65700 1 -- 172.21.15.60:0/1808965751 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb18c10af10 con 0x7fb18c100850 2023-07-20T01:27:16.729 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.751+0000 7fb188ff9700 1 -- 172.21.15.60:0/1808965751 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2020 (secure 0 0 0) 0x7fb18009a820 con 0x7fb18c100850 2023-07-20T01:27:16.729 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:27:16.730 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/3759595107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3759595107},{"type":"v1","addr":"172.21.15.183:6833","nonce":3759595107}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1706932385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1706932385},{"type":"v1","addr":"172.21.15.136:6833","nonce":1706932385}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-07-20T01:27:16.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.753+0000 7fb191f65700 1 -- 172.21.15.60:0/1808965751 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fb1740a1220 msgr2=0x7fb1740a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:16.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.753+0000 7fb191f65700 1 --2- 172.21.15.60:0/1808965751 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fb1740a1220 0x7fb1740a36e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb18c1006b0 tx=0x7fb184009040 comp rx=0 tx=0).stop 2023-07-20T01:27:16.731 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.753+0000 7fb191f65700 1 -- 172.21.15.60:0/1808965751 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18c100850 msgr2=0x7fb18c1968b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:16.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.753+0000 7fb191f65700 1 --2- 172.21.15.60:0/1808965751 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18c100850 0x7fb18c1968b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb180009fd0 tx=0x7fb18000fa00 comp rx=0 tx=0).stop 2023-07-20T01:27:16.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.753+0000 7fb191f65700 1 -- 172.21.15.60:0/1808965751 shutdown_connections 2023-07-20T01:27:16.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.753+0000 7fb191f65700 1 --2- 172.21.15.60:0/1808965751 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fb1740a1220 0x7fb1740a36e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:16.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.753+0000 7fb191f65700 1 --2- 172.21.15.60:0/1808965751 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb18c1011f0 0x7fb18c19d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:16.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.753+0000 7fb191f65700 1 --2- 172.21.15.60:0/1808965751 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb18c0ff660 0x7fb18c196370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:16.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.753+0000 7fb191f65700 1 --2- 172.21.15.60:0/1808965751 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18c100850 0x7fb18c1968b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:16.732 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.753+0000 7fb191f65700 1 -- 172.21.15.60:0/1808965751 >> 172.21.15.60:0/1808965751 conn(0x7fb18c0fab90 msgr2=0x7fb18c068960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:16.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.754+0000 7fb191f65700 1 -- 172.21.15.60:0/1808965751 shutdown_connections 2023-07-20T01:27:16.733 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:16.754+0000 7fb191f65700 1 -- 172.21.15.60:0/1808965751 wait complete. 2023-07-20T01:27:16.733 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 3 2023-07-20T01:27:16.743 INFO:journalctl@ceph.mds.j.smithi060.stdout:Jul 20 01:27:16 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-j[129735]: starting mds.j at 2023-07-20T01:27:16.743 INFO:journalctl@ceph.mds.j.smithi060.stdout:Jul 20 01:27:16 smithi060 bash[129719]: 992076e39ae53424be0610edd84397b9b4bdd6004e3d944f03aede7da3613251 2023-07-20T01:27:16.743 INFO:journalctl@ceph.mds.j.smithi060.stdout:Jul 20 01:27:16 smithi060 systemd[1]: Started Ceph mds.j for 7daf031e-269b-11ee-9b34-001a4aab830c. 2023-07-20T01:27:17.809 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:17 smithi060 ceph-mon[96524]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:17.809 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:17 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:17.809 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:17 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:17.810 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:17 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:17.810 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:17 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:17.810 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:17 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:17.810 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:17 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:17.810 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:17 smithi060 ceph-mon[96524]: Deploying daemon mds.d on smithi060 2023-07-20T01:27:17.811 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:17 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/1808965751' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:17.811 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:17 smithi060 ceph-mon[96524]: mds.? [v2:172.21.15.60:6834/2893530475,v1:172.21.15.60:6835/2893530475] up:boot 2023-07-20T01:27:17.811 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:17 smithi060 ceph-mon[96524]: fsmap 3 up:standby 2023-07-20T01:27:17.811 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:17 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-20T01:27:17.894 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:17 smithi136 ceph-mon[98083]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:17.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:17 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:17.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:17 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:17.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:17 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:17.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:17 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:17.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:17 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:17.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:17 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:17.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:17 smithi136 ceph-mon[98083]: Deploying daemon mds.d on smithi060 2023-07-20T01:27:17.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:17 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/1808965751' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:17.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:17 smithi136 ceph-mon[98083]: mds.? [v2:172.21.15.60:6834/2893530475,v1:172.21.15.60:6835/2893530475] up:boot 2023-07-20T01:27:17.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:17 smithi136 ceph-mon[98083]: fsmap 3 up:standby 2023-07-20T01:27:17.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:17 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-20T01:27:17.947 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:17 smithi183 ceph-mon[98477]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:17.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:17 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:17.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:17 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:17.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:17 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:17.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:17 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:17.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:17 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:17.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:17 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:17.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:17 smithi183 ceph-mon[98477]: Deploying daemon mds.d on smithi060 2023-07-20T01:27:17.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:17 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/1808965751' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:17.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:17 smithi183 ceph-mon[98477]: mds.? [v2:172.21.15.60:6834/2893530475,v1:172.21.15.60:6835/2893530475] up:boot 2023-07-20T01:27:17.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:17 smithi183 ceph-mon[98477]: fsmap 3 up:standby 2023-07-20T01:27:17.950 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:17 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-20T01:27:18.077 INFO:journalctl@ceph.mds.d.smithi060.stdout:Jul 20 01:27:18 smithi060 systemd[1]: Starting Ceph mds.d for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:27:18.620 INFO:journalctl@ceph.mds.d.smithi060.stdout:Jul 20 01:27:18 smithi060 podman[130104]: 2023-07-20T01:27:18.745 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T01:27:18.887 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.909+0000 7f86a6637700 1 -- 172.21.15.60:0/1016583326 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86a00ff650 msgr2=0x7f86a00ffa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:18.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.909+0000 7f86a6637700 1 --2- 172.21.15.60:0/1016583326 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86a00ff650 0x7f86a00ffa70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f86900099c0 tx=0x7f8690009cd0 comp rx=0 tx=0).stop 2023-07-20T01:27:18.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.909+0000 7f86a6637700 1 -- 172.21.15.60:0/1016583326 shutdown_connections 2023-07-20T01:27:18.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.909+0000 7f86a6637700 1 --2- 172.21.15.60:0/1016583326 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86a01011e0 0x7f86a0107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:18.888 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.909+0000 7f86a6637700 1 --2- 172.21.15.60:0/1016583326 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86a00ff650 0x7f86a00ffa70 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:18.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.909+0000 7f86a6637700 1 --2- 172.21.15.60:0/1016583326 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86a0100840 0x7f86a0100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:18.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.909+0000 7f86a6637700 1 -- 172.21.15.60:0/1016583326 >> 172.21.15.60:0/1016583326 conn(0x7f86a00faba0 msgr2=0x7f86a00fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:18.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.909+0000 7f86a6637700 1 -- 172.21.15.60:0/1016583326 shutdown_connections 2023-07-20T01:27:18.889 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.909+0000 7f86a6637700 1 -- 172.21.15.60:0/1016583326 wait complete. 2023-07-20T01:27:18.892 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.910+0000 7f86a6637700 1 Processor -- start 2023-07-20T01:27:18.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.910+0000 7f86a6637700 1 -- start start 2023-07-20T01:27:18.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.911+0000 7f86a6637700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86a00ff650 0x7f86a01952c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:18.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.911+0000 7f86a6637700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86a0100840 0x7f86a0195800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:18.893 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.911+0000 7f86a6637700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86a01011e0 0x7f86a019c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:18.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.911+0000 7f86a6637700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a019ce40 con 0x7f86a0100840 2023-07-20T01:27:18.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.911+0000 7f86a6637700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a019cf80 con 0x7f86a01011e0 2023-07-20T01:27:18.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.911+0000 7f86a6637700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a019d110 con 0x7f86a00ff650 2023-07-20T01:27:18.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.911+0000 7f869f7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86a0100840 0x7f86a0195800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:18.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.911+0000 7f869f7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86a0100840 0x7f86a0195800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42894/0 (socket says 172.21.15.60:42894) 2023-07-20T01:27:18.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.911+0000 7f869f7fe700 1 -- 172.21.15.60:0/2820746687 learned_addr learned my addr 172.21.15.60:0/2820746687 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:27:18.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.911+0000 7f86a4bd4700 1 --2- 172.21.15.60:0/2820746687 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86a01011e0 0x7f86a019c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:18.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.911+0000 7f869ffff700 1 --2- 172.21.15.60:0/2820746687 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86a00ff650 0x7f86a01952c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:18.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.911+0000 7f869f7fe700 1 -- 172.21.15.60:0/2820746687 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86a00ff650 msgr2=0x7f86a01952c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:18.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.911+0000 7f869f7fe700 1 --2- 172.21.15.60:0/2820746687 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86a00ff650 0x7f86a01952c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:18.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.911+0000 7f869f7fe700 1 -- 172.21.15.60:0/2820746687 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86a01011e0 msgr2=0x7f86a019c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:18.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.911+0000 7f869f7fe700 1 --2- 172.21.15.60:0/2820746687 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86a01011e0 0x7f86a019c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:18.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.911+0000 7f869f7fe700 1 -- 172.21.15.60:0/2820746687 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86900095f0 con 0x7f86a0100840 2023-07-20T01:27:18.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.911+0000 7f869f7fe700 1 --2- 172.21.15.60:0/2820746687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86a0100840 0x7f86a0195800 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f869400d950 tx=0x7f869400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:18.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.911+0000 7f869d7fa700 1 -- 172.21.15.60:0/2820746687 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8694009940 con 0x7f86a0100840 2023-07-20T01:27:18.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.912+0000 7f86a6637700 1 -- 172.21.15.60:0/2820746687 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86a019d3f0 con 0x7f86a0100840 2023-07-20T01:27:18.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.912+0000 7f869d7fa700 1 -- 172.21.15.60:0/2820746687 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8694010460 con 0x7f86a0100840 2023-07-20T01:27:18.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.912+0000 7f869d7fa700 1 -- 172.21.15.60:0/2820746687 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869400f620 con 0x7f86a0100840 2023-07-20T01:27:18.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.912+0000 7f86a6637700 1 -- 172.21.15.60:0/2820746687 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86a019d900 con 0x7f86a0100840 2023-07-20T01:27:18.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.913+0000 7f86a6637700 1 -- 172.21.15.60:0/2820746687 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86a010ada0 con 0x7f86a0100840 2023-07-20T01:27:18.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.914+0000 7f869d7fa700 1 -- 172.21.15.60:0/2820746687 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f86940105d0 con 0x7f86a0100840 2023-07-20T01:27:18.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.914+0000 7f869d7fa700 1 --2- 172.21.15.60:0/2820746687 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f86880a11a0 0x7f86880a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:18.897 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.914+0000 7f869ffff700 1 --2- 172.21.15.60:0/2820746687 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f86880a11a0 0x7f86880a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:18.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.914+0000 7f869d7fa700 1 -- 172.21.15.60:0/2820746687 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f86940cc2c0 con 0x7f86a0100840 2023-07-20T01:27:18.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.915+0000 7f869ffff700 1 --2- 172.21.15.60:0/2820746687 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f86880a11a0 0x7f86880a3660 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8690006010 tx=0x7f8690000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:18.898 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:18.919+0000 7f869d7fa700 1 -- 172.21.15.60:0/2820746687 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8694095eb0 con 0x7f86a0100840 2023-07-20T01:27:18.924 INFO:journalctl@ceph.mds.d.smithi060.stdout:Jul 20 01:27:18 smithi060 ceph-mds[130145]: -- 172.21.15.60:0/637270788 <== mon.0 v2:172.21.15.60:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e23141d1e0 con 0x55e2313f2800 2023-07-20T01:27:18.924 INFO:journalctl@ceph.mds.d.smithi060.stdout:Jul 20 01:27:18 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-d[130121]: starting mds.d at 2023-07-20T01:27:18.925 INFO:journalctl@ceph.mds.d.smithi060.stdout:Jul 20 01:27:18 smithi060 bash[130104]: db4e72797250223d548fb96fff16c31887b16b99d2229ce76515921ed0423066 2023-07-20T01:27:18.925 INFO:journalctl@ceph.mds.d.smithi060.stdout:Jul 20 01:27:18 smithi060 systemd[1]: Started Ceph mds.d for 7daf031e-269b-11ee-9b34-001a4aab830c. 2023-07-20T01:27:19.143 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:19.165+0000 7f86a6637700 1 -- 172.21.15.60:0/2820746687 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86a010af90 con 0x7f86a0100840 2023-07-20T01:27:19.144 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:19.166+0000 7f869d7fa700 1 -- 172.21.15.60:0/2820746687 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2777 (secure 0 0 0) 0x7f86940994d0 con 0x7f86a0100840 2023-07-20T01:27:19.144 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:27:19.144 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2893530475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2893530475},{"type":"v1","addr":"172.21.15.60:6835","nonce":2893530475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/3759595107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3759595107},{"type":"v1","addr":"172.21.15.183:6833","nonce":3759595107}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1706932385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1706932385},{"type":"v1","addr":"172.21.15.136:6833","nonce":1706932385}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-07-20T01:27:19.145 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:19.168+0000 7f86a6637700 1 -- 172.21.15.60:0/2820746687 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f86880a11a0 msgr2=0x7f86880a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:19.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:19.168+0000 7f86a6637700 1 --2- 172.21.15.60:0/2820746687 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f86880a11a0 0x7f86880a3660 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8690006010 tx=0x7f8690000bc0 comp rx=0 tx=0).stop 2023-07-20T01:27:19.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:19.168+0000 7f86a6637700 1 -- 172.21.15.60:0/2820746687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86a0100840 msgr2=0x7f86a0195800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:19.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:19.168+0000 7f86a6637700 1 --2- 172.21.15.60:0/2820746687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86a0100840 0x7f86a0195800 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f869400d950 tx=0x7f869400dc60 comp rx=0 tx=0).stop 2023-07-20T01:27:19.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:19.168+0000 7f86a6637700 1 -- 172.21.15.60:0/2820746687 shutdown_connections 2023-07-20T01:27:19.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:19.168+0000 7f86a6637700 1 --2- 172.21.15.60:0/2820746687 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f86880a11a0 0x7f86880a3660 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:19.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:19.168+0000 7f86a6637700 1 --2- 172.21.15.60:0/2820746687 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86a01011e0 0x7f86a019c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:19.146 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:19.168+0000 7f86a6637700 1 --2- 172.21.15.60:0/2820746687 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86a00ff650 0x7f86a01952c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:19.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:19.168+0000 7f86a6637700 1 --2- 172.21.15.60:0/2820746687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86a0100840 0x7f86a0195800 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:19.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:19.168+0000 7f86a6637700 1 -- 172.21.15.60:0/2820746687 >> 172.21.15.60:0/2820746687 conn(0x7f86a00faba0 msgr2=0x7f86a00fced0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:19.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:19.168+0000 7f86a6637700 1 -- 172.21.15.60:0/2820746687 shutdown_connections 2023-07-20T01:27:19.147 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:19.168+0000 7f86a6637700 1 -- 172.21.15.60:0/2820746687 wait complete. 2023-07-20T01:27:19.147 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 4 2023-07-20T01:27:20.047 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:19 smithi136 ceph-mon[98083]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:20.047 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:19 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:20.048 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:19 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:20.048 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:19 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:20.048 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:19 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:20.048 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:19 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:20.049 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:19 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:20.049 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:19 smithi136 ceph-mon[98083]: Deploying daemon mds.e on smithi136 2023-07-20T01:27:20.049 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:19 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/2820746687' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:20.049 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:19 smithi136 ceph-mon[98083]: mds.? [v2:172.21.15.60:6836/302154574,v1:172.21.15.60:6837/302154574] up:boot 2023-07-20T01:27:20.050 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:19 smithi136 ceph-mon[98083]: fsmap 4 up:standby 2023-07-20T01:27:20.050 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:19 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-20T01:27:20.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:19 smithi060 ceph-mon[96524]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:20.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:19 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:20.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:19 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:20.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:19 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:20.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:19 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:20.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:19 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:20.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:19 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:20.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:19 smithi060 ceph-mon[96524]: Deploying daemon mds.e on smithi136 2023-07-20T01:27:20.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:19 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/2820746687' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:20.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:19 smithi060 ceph-mon[96524]: mds.? [v2:172.21.15.60:6836/302154574,v1:172.21.15.60:6837/302154574] up:boot 2023-07-20T01:27:20.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:19 smithi060 ceph-mon[96524]: fsmap 4 up:standby 2023-07-20T01:27:20.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:19 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-20T01:27:20.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:19 smithi183 ceph-mon[98477]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:20.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:19 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:20.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:19 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:20.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:19 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:20.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:19 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:20.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:19 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:20.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:19 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:20.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:19 smithi183 ceph-mon[98477]: Deploying daemon mds.e on smithi136 2023-07-20T01:27:20.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:19 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/2820746687' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:20.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:19 smithi183 ceph-mon[98477]: mds.? [v2:172.21.15.60:6836/302154574,v1:172.21.15.60:6837/302154574] up:boot 2023-07-20T01:27:20.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:19 smithi183 ceph-mon[98477]: fsmap 4 up:standby 2023-07-20T01:27:20.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:19 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-20T01:27:20.645 INFO:journalctl@ceph.mds.e.smithi136.stdout:Jul 20 01:27:20 smithi136 systemd[1]: Starting Ceph mds.e for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:27:20.936 INFO:journalctl@ceph.mds.e.smithi136.stdout:Jul 20 01:27:20 smithi136 podman[122464]: 2023-07-20T01:27:21.159 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T01:27:21.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.323+0000 7f81b6240700 1 -- 172.21.15.60:0/285233817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81b0103b00 msgr2=0x7f81b0105ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:21.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.323+0000 7f81b6240700 1 --2- 172.21.15.60:0/285233817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81b0103b00 0x7f81b0105ef0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f81a4009990 tx=0x7f81a4009ca0 comp rx=0 tx=0).stop 2023-07-20T01:27:21.301 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.323+0000 7f81b6240700 1 -- 172.21.15.60:0/285233817 shutdown_connections 2023-07-20T01:27:21.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.323+0000 7f81b6240700 1 --2- 172.21.15.60:0/285233817 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f81b0106430 0x7f81b01088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:21.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.323+0000 7f81b6240700 1 --2- 172.21.15.60:0/285233817 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f81b01011d0 0x7f81b01035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:21.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.323+0000 7f81b6240700 1 --2- 172.21.15.60:0/285233817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81b0103b00 0x7f81b0105ef0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:21.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.323+0000 7f81b6240700 1 -- 172.21.15.60:0/285233817 >> 172.21.15.60:0/285233817 conn(0x7f81b00faba0 msgr2=0x7f81b00fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:21.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.323+0000 7f81b6240700 1 -- 172.21.15.60:0/285233817 shutdown_connections 2023-07-20T01:27:21.302 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.323+0000 7f81b6240700 1 -- 172.21.15.60:0/285233817 wait complete. 2023-07-20T01:27:21.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.324+0000 7f81b6240700 1 Processor -- start 2023-07-20T01:27:21.304 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.324+0000 7f81b6240700 1 -- start start 2023-07-20T01:27:21.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.324+0000 7f81b6240700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f81b01011d0 0x7f81b0100b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:21.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.325+0000 7f81b6240700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f81b0103b00 0x7f81b00ff180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:21.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.325+0000 7f81b6240700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81b0106430 0x7f81b00ff6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:21.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.325+0000 7f81b6240700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b00ffe60 con 0x7f81b0106430 2023-07-20T01:27:21.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.325+0000 7f81b6240700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b00fffd0 con 0x7f81b0103b00 2023-07-20T01:27:21.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.325+0000 7f81b6240700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b01a03f0 con 0x7f81b01011d0 2023-07-20T01:27:21.305 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.325+0000 7f81affff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81b0106430 0x7f81b00ff6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:21.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.325+0000 7f81affff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81b0106430 0x7f81b00ff6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42912/0 (socket says 172.21.15.60:42912) 2023-07-20T01:27:21.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.325+0000 7f81affff700 1 -- 172.21.15.60:0/799583245 learned_addr learned my addr 172.21.15.60:0/799583245 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:27:21.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.325+0000 7f81af7fe700 1 --2- 172.21.15.60:0/799583245 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f81b01011d0 0x7f81b0100b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:21.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.325+0000 7f81aeffd700 1 --2- 172.21.15.60:0/799583245 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f81b0103b00 0x7f81b00ff180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:21.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.325+0000 7f81affff700 1 -- 172.21.15.60:0/799583245 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f81b01011d0 msgr2=0x7f81b0100b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:21.306 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.325+0000 7f81affff700 1 --2- 172.21.15.60:0/799583245 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f81b01011d0 0x7f81b0100b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:21.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.325+0000 7f81affff700 1 -- 172.21.15.60:0/799583245 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f81b0103b00 msgr2=0x7f81b00ff180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:21.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.325+0000 7f81affff700 1 --2- 172.21.15.60:0/799583245 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f81b0103b00 0x7f81b00ff180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:21.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.325+0000 7f81affff700 1 -- 172.21.15.60:0/799583245 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a40095f0 con 0x7f81b0106430 2023-07-20T01:27:21.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.325+0000 7f81af7fe700 1 --2- 172.21.15.60:0/799583245 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f81b01011d0 0x7f81b0100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T01:27:21.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.325+0000 7f81affff700 1 --2- 172.21.15.60:0/799583245 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81b0106430 0x7f81b00ff6c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f81a8007ee0 tx=0x7f81a800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:21.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.325+0000 7f81acff9700 1 -- 172.21.15.60:0/799583245 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a8017070 con 0x7f81b0106430 2023-07-20T01:27:21.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.325+0000 7f81acff9700 1 -- 172.21.15.60:0/799583245 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f81a800f040 con 0x7f81b0106430 2023-07-20T01:27:21.307 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.325+0000 7f81b6240700 1 -- 172.21.15.60:0/799583245 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81b01a06d0 con 0x7f81b0106430 2023-07-20T01:27:21.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.325+0000 7f81acff9700 1 -- 172.21.15.60:0/799583245 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a8005570 con 0x7f81b0106430 2023-07-20T01:27:21.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.326+0000 7f81b6240700 1 -- 172.21.15.60:0/799583245 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81b01a0be0 con 0x7f81b0106430 2023-07-20T01:27:21.308 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.326+0000 7f81b6240700 1 -- 172.21.15.60:0/799583245 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81b00fc720 con 0x7f81b0106430 2023-07-20T01:27:21.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.328+0000 7f81acff9700 1 -- 172.21.15.60:0/799583245 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f81a800db20 con 0x7f81b0106430 2023-07-20T01:27:21.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.328+0000 7f81acff9700 1 --2- 172.21.15.60:0/799583245 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f81980a1220 0x7f81980a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:21.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.328+0000 7f81acff9700 1 -- 172.21.15.60:0/799583245 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f81a80cbb90 con 0x7f81b0106430 2023-07-20T01:27:21.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.328+0000 7f81af7fe700 1 --2- 172.21.15.60:0/799583245 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f81980a1220 0x7f81980a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:21.309 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.329+0000 7f81af7fe700 1 --2- 172.21.15.60:0/799583245 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f81980a1220 0x7f81980a36e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f81a0011490 tx=0x7f81a0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:21.310 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.332+0000 7f81acff9700 1 -- 172.21.15.60:0/799583245 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f81a8095780 con 0x7f81b0106430 2023-07-20T01:27:21.394 INFO:journalctl@ceph.mds.e.smithi136.stdout:Jul 20 01:27:20 smithi136 ceph-mds[122505]: -- 172.21.15.136:0/4279440094 <== mon.2 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558d8b9f11e0 con 0x558d8b9c6800 2023-07-20T01:27:21.394 INFO:journalctl@ceph.mds.e.smithi136.stdout:Jul 20 01:27:20 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-e[122481]: starting mds.e at 2023-07-20T01:27:21.395 INFO:journalctl@ceph.mds.e.smithi136.stdout:Jul 20 01:27:20 smithi136 bash[122464]: 7731e687b3e03e86a0d7be20af7202b7cf3cc6415223db5f5503535a874d4f0d 2023-07-20T01:27:21.395 INFO:journalctl@ceph.mds.e.smithi136.stdout:Jul 20 01:27:21 smithi136 systemd[1]: Started Ceph mds.e for 7daf031e-269b-11ee-9b34-001a4aab830c. 2023-07-20T01:27:21.557 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.579+0000 7f81b6240700 1 -- 172.21.15.60:0/799583245 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81b002c970 con 0x7f81b0106430 2023-07-20T01:27:21.558 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.580+0000 7f81acff9700 1 -- 172.21.15.60:0/799583245 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3531 (secure 0 0 0) 0x7f81a8098da0 con 0x7f81b0106430 2023-07-20T01:27:21.558 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:27:21.558 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2893530475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2893530475},{"type":"v1","addr":"172.21.15.60:6835","nonce":2893530475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/302154574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":302154574},{"type":"v1","addr":"172.21.15.60:6837","nonce":302154574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/3759595107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3759595107},{"type":"v1","addr":"172.21.15.183:6833","nonce":3759595107}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1706932385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1706932385},{"type":"v1","addr":"172.21.15.136:6833","nonce":1706932385}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-07-20T01:27:21.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.582+0000 7f81b6240700 1 -- 172.21.15.60:0/799583245 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f81980a1220 msgr2=0x7f81980a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:21.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.582+0000 7f81b6240700 1 --2- 172.21.15.60:0/799583245 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f81980a1220 0x7f81980a36e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f81a0011490 tx=0x7f81a0009260 comp rx=0 tx=0).stop 2023-07-20T01:27:21.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.582+0000 7f81b6240700 1 -- 172.21.15.60:0/799583245 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81b0106430 msgr2=0x7f81b00ff6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:21.560 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.582+0000 7f81b6240700 1 --2- 172.21.15.60:0/799583245 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81b0106430 0x7f81b00ff6c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f81a8007ee0 tx=0x7f81a800d3b0 comp rx=0 tx=0).stop 2023-07-20T01:27:21.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.582+0000 7f81b6240700 1 -- 172.21.15.60:0/799583245 shutdown_connections 2023-07-20T01:27:21.561 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.582+0000 7f81b6240700 1 --2- 172.21.15.60:0/799583245 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f81980a1220 0x7f81980a36e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:21.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.582+0000 7f81b6240700 1 --2- 172.21.15.60:0/799583245 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81b0106430 0x7f81b00ff6c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:21.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.582+0000 7f81b6240700 1 --2- 172.21.15.60:0/799583245 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f81b01011d0 0x7f81b0100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:21.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.582+0000 7f81b6240700 1 --2- 172.21.15.60:0/799583245 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f81b0103b00 0x7f81b00ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:21.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.582+0000 7f81b6240700 1 -- 172.21.15.60:0/799583245 >> 172.21.15.60:0/799583245 conn(0x7f81b00faba0 msgr2=0x7f81b0101e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:21.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.582+0000 7f81b6240700 1 -- 172.21.15.60:0/799583245 shutdown_connections 2023-07-20T01:27:21.562 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:21.582+0000 7f81b6240700 1 -- 172.21.15.60:0/799583245 wait complete. 2023-07-20T01:27:21.563 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 5 2023-07-20T01:27:21.924 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:21 smithi060 ceph-mon[96524]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:21.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:21 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:21.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:21 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:21.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:21 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:21.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:21 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:21.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:21 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:21.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:21 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:21.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:21 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/799583245' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:21.980 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:21 smithi183 ceph-mon[98477]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:21.980 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:21 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:21.980 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:21 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:21.981 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:21 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:21.981 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:21 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:21.981 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:21 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:21.981 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:21 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:21.981 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:21 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/799583245' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:22.144 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:21 smithi136 ceph-mon[98083]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:22.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:21 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:22.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:21 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:22.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:21 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:22.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:21 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:22.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:21 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:22.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:21 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:22.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:21 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/799583245' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:22.925 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:22 smithi183 ceph-mon[98477]: Deploying daemon mds.f on smithi183 2023-07-20T01:27:22.926 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:22 smithi183 ceph-mon[98477]: mds.? [v2:172.21.15.136:6834/3349527335,v1:172.21.15.136:6835/3349527335] up:boot 2023-07-20T01:27:22.926 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:22 smithi183 ceph-mon[98477]: fsmap 5 up:standby 2023-07-20T01:27:22.926 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:22 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-20T01:27:22.926 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 20 01:27:22 smithi183 systemd[1]: Starting Ceph mds.f for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:27:23.144 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:22 smithi136 ceph-mon[98083]: Deploying daemon mds.f on smithi183 2023-07-20T01:27:23.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:22 smithi136 ceph-mon[98083]: mds.? [v2:172.21.15.136:6834/3349527335,v1:172.21.15.136:6835/3349527335] up:boot 2023-07-20T01:27:23.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:22 smithi136 ceph-mon[98083]: fsmap 5 up:standby 2023-07-20T01:27:23.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:22 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-20T01:27:23.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:22 smithi060 ceph-mon[96524]: Deploying daemon mds.f on smithi183 2023-07-20T01:27:23.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:22 smithi060 ceph-mon[96524]: mds.? [v2:172.21.15.136:6834/3349527335,v1:172.21.15.136:6835/3349527335] up:boot 2023-07-20T01:27:23.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:22 smithi060 ceph-mon[96524]: fsmap 5 up:standby 2023-07-20T01:27:23.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:22 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-20T01:27:23.185 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 20 01:27:22 smithi183 podman[123826]: 2023-07-20T01:27:23.448 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 20 01:27:23 smithi183 ceph-mds[123866]: -- 172.21.15.183:0/2871185689 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563df7bb31e0 con 0x563df7b88800 2023-07-20T01:27:23.448 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 20 01:27:23 smithi183 bash[123826]: 3e99be003742b027ea426af3ae861a3a904e519a4707ebdb9bf422e25122fab3 2023-07-20T01:27:23.448 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 20 01:27:23 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-f[123842]: starting mds.f at 2023-07-20T01:27:23.449 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 20 01:27:23 smithi183 systemd[1]: Started Ceph mds.f for 7daf031e-269b-11ee-9b34-001a4aab830c. 2023-07-20T01:27:23.573 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T01:27:23.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.735+0000 7f17b53c9700 1 -- 172.21.15.60:0/2371608770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17b00ff650 msgr2=0x7f17b00ffa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:23.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.735+0000 7f17b53c9700 1 --2- 172.21.15.60:0/2371608770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17b00ff650 0x7f17b00ffa70 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f17a00099c0 tx=0x7f17a0009cd0 comp rx=0 tx=0).stop 2023-07-20T01:27:23.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.735+0000 7f17b53c9700 1 -- 172.21.15.60:0/2371608770 shutdown_connections 2023-07-20T01:27:23.713 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.735+0000 7f17b53c9700 1 --2- 172.21.15.60:0/2371608770 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f17b01011e0 0x7f17b0107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:23.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.735+0000 7f17b53c9700 1 --2- 172.21.15.60:0/2371608770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17b00ff650 0x7f17b00ffa70 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:23.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.735+0000 7f17b53c9700 1 --2- 172.21.15.60:0/2371608770 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17b0100840 0x7f17b0100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:23.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.735+0000 7f17b53c9700 1 -- 172.21.15.60:0/2371608770 >> 172.21.15.60:0/2371608770 conn(0x7f17b00faba0 msgr2=0x7f17b00fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:23.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.735+0000 7f17b53c9700 1 -- 172.21.15.60:0/2371608770 shutdown_connections 2023-07-20T01:27:23.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.735+0000 7f17b53c9700 1 -- 172.21.15.60:0/2371608770 wait complete. 2023-07-20T01:27:23.714 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.735+0000 7f17b53c9700 1 Processor -- start 2023-07-20T01:27:23.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.736+0000 7f17b53c9700 1 -- start start 2023-07-20T01:27:23.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.736+0000 7f17b53c9700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17b00ff650 0x7f17b01962f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:23.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.736+0000 7f17b53c9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17b0100840 0x7f17b0196830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:23.716 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.736+0000 7f17b53c9700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f17b01011e0 0x7f17b019d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:23.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.736+0000 7f17b53c9700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b019de70 con 0x7f17b0100840 2023-07-20T01:27:23.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.736+0000 7f17b53c9700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b019dfb0 con 0x7f17b01011e0 2023-07-20T01:27:23.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.736+0000 7f17b53c9700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b019e140 con 0x7f17b00ff650 2023-07-20T01:27:23.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.736+0000 7f17ae7fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17b0100840 0x7f17b0196830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:23.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.736+0000 7f17aeffd700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17b00ff650 0x7f17b01962f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:23.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.736+0000 7f17aeffd700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17b00ff650 0x7f17b01962f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.60:35912/0 (socket says 172.21.15.60:35912) 2023-07-20T01:27:23.717 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.736+0000 7f17ae7fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17b0100840 0x7f17b0196830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42924/0 (socket says 172.21.15.60:42924) 2023-07-20T01:27:23.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.736+0000 7f17aeffd700 1 -- 172.21.15.60:0/3241842367 learned_addr learned my addr 172.21.15.60:0/3241842367 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:27:23.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.736+0000 7f17af7fe700 1 --2- 172.21.15.60:0/3241842367 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f17b01011e0 0x7f17b019d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:23.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.736+0000 7f17aeffd700 1 -- 172.21.15.60:0/3241842367 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f17b01011e0 msgr2=0x7f17b019d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:23.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.736+0000 7f17aeffd700 1 --2- 172.21.15.60:0/3241842367 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f17b01011e0 0x7f17b019d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:23.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.736+0000 7f17aeffd700 1 -- 172.21.15.60:0/3241842367 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17b0100840 msgr2=0x7f17b0196830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:23.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.736+0000 7f17aeffd700 1 --2- 172.21.15.60:0/3241842367 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17b0100840 0x7f17b0196830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:23.718 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.736+0000 7f17aeffd700 1 -- 172.21.15.60:0/3241842367 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a00095f0 con 0x7f17b00ff650 2023-07-20T01:27:23.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.736+0000 7f17ae7fc700 1 --2- 172.21.15.60:0/3241842367 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17b0100840 0x7f17b0196830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T01:27:23.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.737+0000 7f17aeffd700 1 --2- 172.21.15.60:0/3241842367 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17b00ff650 0x7f17b01962f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f17a0009f50 tx=0x7f17a0003a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:23.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.737+0000 7f17a7fff700 1 -- 172.21.15.60:0/3241842367 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a001d070 con 0x7f17b00ff650 2023-07-20T01:27:23.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.737+0000 7f17b53c9700 1 -- 172.21.15.60:0/3241842367 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17b019e3c0 con 0x7f17b00ff650 2023-07-20T01:27:23.719 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.737+0000 7f17a7fff700 1 -- 172.21.15.60:0/3241842367 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f17a000bda0 con 0x7f17b00ff650 2023-07-20T01:27:23.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.737+0000 7f17a7fff700 1 -- 172.21.15.60:0/3241842367 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a0022c80 con 0x7f17b00ff650 2023-07-20T01:27:23.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.737+0000 7f17b53c9700 1 -- 172.21.15.60:0/3241842367 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17b019e8c0 con 0x7f17b00ff650 2023-07-20T01:27:23.720 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.738+0000 7f17b53c9700 1 -- 172.21.15.60:0/3241842367 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17b010ad20 con 0x7f17b00ff650 2023-07-20T01:27:23.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.739+0000 7f17a7fff700 1 -- 172.21.15.60:0/3241842367 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f17a0003f40 con 0x7f17b00ff650 2023-07-20T01:27:23.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.740+0000 7f17a7fff700 1 --2- 172.21.15.60:0/3241842367 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f179c0a11a0 0x7f179c0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:23.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.740+0000 7f17ae7fc700 1 --2- 172.21.15.60:0/3241842367 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f179c0a11a0 0x7f179c0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:23.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.740+0000 7f17a7fff700 1 -- 172.21.15.60:0/3241842367 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f17a00ce090 con 0x7f17b00ff650 2023-07-20T01:27:23.721 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.740+0000 7f17ae7fc700 1 --2- 172.21.15.60:0/3241842367 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f179c0a11a0 0x7f179c0a3660 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1798005950 tx=0x7f179800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:23.722 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.743+0000 7f17a7fff700 1 -- 172.21.15.60:0/3241842367 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f17a0097cb0 con 0x7f17b00ff650 2023-07-20T01:27:23.924 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:23 smithi060 ceph-mon[96524]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:23.924 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:23 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:23.924 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:23 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:23.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:23 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:23.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:23 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:23.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:23 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:23.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:23 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:23.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.989+0000 7f17b53c9700 1 -- 172.21.15.60:0/3241842367 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17b019ef00 con 0x7f17b00ff650 2023-07-20T01:27:23.967 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.990+0000 7f17a7fff700 1 -- 172.21.15.60:0/3241842367 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5051 (secure 0 0 0) 0x7f17a009b2d0 con 0x7f17b00ff650 2023-07-20T01:27:23.968 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:27:23.968 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2893530475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2893530475},{"type":"v1","addr":"172.21.15.60:6835","nonce":2893530475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/302154574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":302154574},{"type":"v1","addr":"172.21.15.60:6837","nonce":302154574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/3759595107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3759595107},{"type":"v1","addr":"172.21.15.183:6833","nonce":3759595107}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1706932385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1706932385},{"type":"v1","addr":"172.21.15.136:6833","nonce":1706932385}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3379647260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3379647260},{"type":"v1","addr":"172.21.15.183:6835","nonce":3379647260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3349527335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3349527335},{"type":"v1","addr":"172.21.15.136:6835","nonce":3349527335}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2023-07-20T01:27:23.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.992+0000 7f17b53c9700 1 -- 172.21.15.60:0/3241842367 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f179c0a11a0 msgr2=0x7f179c0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:23.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.992+0000 7f17b53c9700 1 --2- 172.21.15.60:0/3241842367 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f179c0a11a0 0x7f179c0a3660 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1798005950 tx=0x7f179800b410 comp rx=0 tx=0).stop 2023-07-20T01:27:23.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.992+0000 7f17b53c9700 1 -- 172.21.15.60:0/3241842367 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17b00ff650 msgr2=0x7f17b01962f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:23.970 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.992+0000 7f17b53c9700 1 --2- 172.21.15.60:0/3241842367 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17b00ff650 0x7f17b01962f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f17a0009f50 tx=0x7f17a0003a50 comp rx=0 tx=0).stop 2023-07-20T01:27:23.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.992+0000 7f17b53c9700 1 -- 172.21.15.60:0/3241842367 shutdown_connections 2023-07-20T01:27:23.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.992+0000 7f17b53c9700 1 --2- 172.21.15.60:0/3241842367 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f179c0a11a0 0x7f179c0a3660 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:23.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.992+0000 7f17b53c9700 1 --2- 172.21.15.60:0/3241842367 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f17b01011e0 0x7f17b019d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:23.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.992+0000 7f17b53c9700 1 --2- 172.21.15.60:0/3241842367 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17b00ff650 0x7f17b01962f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:23.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.992+0000 7f17b53c9700 1 --2- 172.21.15.60:0/3241842367 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17b0100840 0x7f17b0196830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:23.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.992+0000 7f17b53c9700 1 -- 172.21.15.60:0/3241842367 >> 172.21.15.60:0/3241842367 conn(0x7f17b00faba0 msgr2=0x7f17b0067990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:23.971 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.992+0000 7f17b53c9700 1 -- 172.21.15.60:0/3241842367 shutdown_connections 2023-07-20T01:27:23.972 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:23.992+0000 7f17b53c9700 1 -- 172.21.15.60:0/3241842367 wait complete. 2023-07-20T01:27:23.972 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 7 2023-07-20T01:27:23.989 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:23 smithi183 ceph-mon[98477]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:23.989 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:23 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:23.990 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:23 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:23.990 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:23 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:23.990 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:23 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:23.990 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:23 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:23.990 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:23 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:24.144 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:23 smithi136 ceph-mon[98083]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:24.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:23 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:24.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:23 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:24.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:23 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:24.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:23 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:24.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:23 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:24.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:23 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:24.802 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:24 smithi183 ceph-mon[98477]: Deploying daemon mds.c on smithi183 2023-07-20T01:27:24.803 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:24 smithi183 ceph-mon[98477]: mds.? [v2:172.21.15.183:6834/3379647260,v1:172.21.15.183:6835/3379647260] up:boot 2023-07-20T01:27:24.803 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:24 smithi183 ceph-mon[98477]: fsmap 6 up:standby 2023-07-20T01:27:24.803 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:24 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-20T01:27:24.803 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:24 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/3241842367' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:25.074 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 20 01:27:24 smithi183 systemd[1]: Starting Ceph mds.c for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:27:25.144 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:24 smithi136 ceph-mon[98083]: Deploying daemon mds.c on smithi183 2023-07-20T01:27:25.144 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:24 smithi136 ceph-mon[98083]: mds.? [v2:172.21.15.183:6834/3379647260,v1:172.21.15.183:6835/3379647260] up:boot 2023-07-20T01:27:25.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:24 smithi136 ceph-mon[98083]: fsmap 6 up:standby 2023-07-20T01:27:25.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:24 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-20T01:27:25.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:24 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/3241842367' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:25.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:24 smithi060 ceph-mon[96524]: Deploying daemon mds.c on smithi183 2023-07-20T01:27:25.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:24 smithi060 ceph-mon[96524]: mds.? [v2:172.21.15.183:6834/3379647260,v1:172.21.15.183:6835/3379647260] up:boot 2023-07-20T01:27:25.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:24 smithi060 ceph-mon[96524]: fsmap 6 up:standby 2023-07-20T01:27:25.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:24 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-20T01:27:25.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:24 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/3241842367' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:25.327 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 20 01:27:25 smithi183 podman[124168]: 2023-07-20T01:27:25.328 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 20 01:27:25 smithi183 ceph-mds[124208]: -- 172.21.15.183:0/3902915386 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f67799d1e0 con 0x55f677972c00 2023-07-20T01:27:25.328 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 20 01:27:25 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-c[124184]: starting mds.c at 2023-07-20T01:27:25.697 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 20 01:27:25 smithi183 bash[124168]: d34cc07b563a5098f19c81ea12230a7f8ad03a212187a376c1a63b1383fab9f3 2023-07-20T01:27:25.697 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 20 01:27:25 smithi183 systemd[1]: Started Ceph mds.c for 7daf031e-269b-11ee-9b34-001a4aab830c. 2023-07-20T01:27:25.983 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T01:27:26.014 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:25 smithi060 ceph-mon[96524]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:26.014 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:25 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:26.014 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:25 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:26.015 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:25 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:26.015 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:25 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:26.015 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:25 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:26.015 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:25 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:26.101 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:25 smithi136 ceph-mon[98083]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:26.102 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:26.102 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:26.102 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:26.102 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:26.102 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:26.102 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:25 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:26.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.146+0000 7ff470ae2700 1 -- 172.21.15.60:0/838356617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff46c100840 msgr2=0x7ff46c100ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:26.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.146+0000 7ff470ae2700 1 --2- 172.21.15.60:0/838356617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff46c100840 0x7ff46c100ca0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff4540099c0 tx=0x7ff454009cd0 comp rx=0 tx=0).stop 2023-07-20T01:27:26.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.146+0000 7ff470ae2700 1 -- 172.21.15.60:0/838356617 shutdown_connections 2023-07-20T01:27:26.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.146+0000 7ff470ae2700 1 --2- 172.21.15.60:0/838356617 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff46c1011e0 0x7ff46c107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:26.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.146+0000 7ff470ae2700 1 --2- 172.21.15.60:0/838356617 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff46c0ff650 0x7ff46c0ffa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:26.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.146+0000 7ff470ae2700 1 --2- 172.21.15.60:0/838356617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff46c100840 0x7ff46c100ca0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:26.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.146+0000 7ff470ae2700 1 -- 172.21.15.60:0/838356617 >> 172.21.15.60:0/838356617 conn(0x7ff46c0faba0 msgr2=0x7ff46c0fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:26.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.146+0000 7ff470ae2700 1 -- 172.21.15.60:0/838356617 shutdown_connections 2023-07-20T01:27:26.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.146+0000 7ff470ae2700 1 -- 172.21.15.60:0/838356617 wait complete. 2023-07-20T01:27:26.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.147+0000 7ff470ae2700 1 Processor -- start 2023-07-20T01:27:26.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.147+0000 7ff470ae2700 1 -- start start 2023-07-20T01:27:26.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.147+0000 7ff470ae2700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff46c0ff650 0x7ff46c1950f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:26.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.147+0000 7ff470ae2700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff46c100840 0x7ff46c195630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:26.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.147+0000 7ff470ae2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff46c1011e0 0x7ff46c19c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:26.127 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.147+0000 7ff470ae2700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff46c19cc70 con 0x7ff46c1011e0 2023-07-20T01:27:26.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.147+0000 7ff470ae2700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff46c19cdb0 con 0x7ff46c100840 2023-07-20T01:27:26.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.147+0000 7ff470ae2700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff46c19cf40 con 0x7ff46c0ff650 2023-07-20T01:27:26.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.147+0000 7ff46ad9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff46c1011e0 0x7ff46c19c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:26.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.147+0000 7ff46ad9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff46c1011e0 0x7ff46c19c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42940/0 (socket says 172.21.15.60:42940) 2023-07-20T01:27:26.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.147+0000 7ff46ad9d700 1 -- 172.21.15.60:0/1025701861 learned_addr learned my addr 172.21.15.60:0/1025701861 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:27:26.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.147+0000 7ff469d9b700 1 --2- 172.21.15.60:0/1025701861 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff46c100840 0x7ff46c195630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:26.128 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.147+0000 7ff46a59c700 1 --2- 172.21.15.60:0/1025701861 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff46c0ff650 0x7ff46c1950f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:26.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.148+0000 7ff46ad9d700 1 -- 172.21.15.60:0/1025701861 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff46c0ff650 msgr2=0x7ff46c1950f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:26.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.148+0000 7ff46ad9d700 1 --2- 172.21.15.60:0/1025701861 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff46c0ff650 0x7ff46c1950f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:26.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.148+0000 7ff46ad9d700 1 -- 172.21.15.60:0/1025701861 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff46c100840 msgr2=0x7ff46c195630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:26.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.148+0000 7ff46ad9d700 1 --2- 172.21.15.60:0/1025701861 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff46c100840 0x7ff46c195630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:26.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.148+0000 7ff46ad9d700 1 -- 172.21.15.60:0/1025701861 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4540095f0 con 0x7ff46c1011e0 2023-07-20T01:27:26.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.148+0000 7ff46ad9d700 1 --2- 172.21.15.60:0/1025701861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff46c1011e0 0x7ff46c19c6a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff464007d70 tx=0x7ff46400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:26.129 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.148+0000 7ff4637fe700 1 -- 172.21.15.60:0/1025701861 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46400ac90 con 0x7ff46c1011e0 2023-07-20T01:27:26.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.148+0000 7ff4637fe700 1 -- 172.21.15.60:0/1025701861 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff46400adf0 con 0x7ff46c1011e0 2023-07-20T01:27:26.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.148+0000 7ff470ae2700 1 -- 172.21.15.60:0/1025701861 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff46c19d220 con 0x7ff46c1011e0 2023-07-20T01:27:26.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.148+0000 7ff4637fe700 1 -- 172.21.15.60:0/1025701861 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4640115e0 con 0x7ff46c1011e0 2023-07-20T01:27:26.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.148+0000 7ff470ae2700 1 -- 172.21.15.60:0/1025701861 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff46c19d730 con 0x7ff46c1011e0 2023-07-20T01:27:26.130 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.149+0000 7ff470ae2700 1 -- 172.21.15.60:0/1025701861 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff46c10ad10 con 0x7ff46c1011e0 2023-07-20T01:27:26.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.150+0000 7ff4637fe700 1 -- 172.21.15.60:0/1025701861 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff464012420 con 0x7ff46c1011e0 2023-07-20T01:27:26.131 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.151+0000 7ff4637fe700 1 --2- 172.21.15.60:0/1025701861 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7ff4580a1220 0x7ff4580a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:26.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.151+0000 7ff46a59c700 1 --2- 172.21.15.60:0/1025701861 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7ff4580a1220 0x7ff4580a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:26.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.151+0000 7ff4637fe700 1 -- 172.21.15.60:0/1025701861 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff4640cc600 con 0x7ff46c1011e0 2023-07-20T01:27:26.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.151+0000 7ff46a59c700 1 --2- 172.21.15.60:0/1025701861 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7ff4580a1220 0x7ff4580a36e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff46c1006a0 tx=0x7ff45c00a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:26.132 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.155+0000 7ff4637fe700 1 -- 172.21.15.60:0/1025701861 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff4640961f0 con 0x7ff46c1011e0 2023-07-20T01:27:26.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:25 smithi183 ceph-mon[98477]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:26.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:25 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:26.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:25 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:26.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:25 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:26.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:25 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:26.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:25 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:26.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:25 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:26.380 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.402+0000 7ff470ae2700 1 -- 172.21.15.60:0/1025701861 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff46c10af00 con 0x7ff46c1011e0 2023-07-20T01:27:26.381 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.403+0000 7ff4637fe700 1 -- 172.21.15.60:0/1025701861 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+5811 (secure 0 0 0) 0x7ff464099810 con 0x7ff46c1011e0 2023-07-20T01:27:26.381 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:27:26.381 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2893530475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2893530475},{"type":"v1","addr":"172.21.15.60:6835","nonce":2893530475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/302154574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":302154574},{"type":"v1","addr":"172.21.15.60:6837","nonce":302154574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/3759595107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3759595107},{"type":"v1","addr":"172.21.15.183:6833","nonce":3759595107}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1706932385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1706932385},{"type":"v1","addr":"172.21.15.136:6833","nonce":1706932385}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3379647260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3379647260},{"type":"v1","addr":"172.21.15.183:6835","nonce":3379647260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3349527335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3349527335},{"type":"v1","addr":"172.21.15.136:6835","nonce":3349527335}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/3540765285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":3540765285},{"type":"v1","addr":"172.21.15.183:6837","nonce":3540765285}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2023-07-20T01:27:26.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.405+0000 7ff470ae2700 1 -- 172.21.15.60:0/1025701861 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7ff4580a1220 msgr2=0x7ff4580a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:26.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.405+0000 7ff470ae2700 1 --2- 172.21.15.60:0/1025701861 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7ff4580a1220 0x7ff4580a36e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff46c1006a0 tx=0x7ff45c00a3b0 comp rx=0 tx=0).stop 2023-07-20T01:27:26.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.405+0000 7ff470ae2700 1 -- 172.21.15.60:0/1025701861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff46c1011e0 msgr2=0x7ff46c19c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:26.383 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.405+0000 7ff470ae2700 1 --2- 172.21.15.60:0/1025701861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff46c1011e0 0x7ff46c19c6a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff464007d70 tx=0x7ff46400a340 comp rx=0 tx=0).stop 2023-07-20T01:27:26.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.405+0000 7ff470ae2700 1 -- 172.21.15.60:0/1025701861 shutdown_connections 2023-07-20T01:27:26.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.405+0000 7ff470ae2700 1 --2- 172.21.15.60:0/1025701861 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7ff4580a1220 0x7ff4580a36e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:26.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.405+0000 7ff470ae2700 1 --2- 172.21.15.60:0/1025701861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff46c1011e0 0x7ff46c19c6a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:26.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.405+0000 7ff470ae2700 1 --2- 172.21.15.60:0/1025701861 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff46c0ff650 0x7ff46c1950f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:26.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.405+0000 7ff470ae2700 1 --2- 172.21.15.60:0/1025701861 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff46c100840 0x7ff46c195630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:26.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.405+0000 7ff470ae2700 1 -- 172.21.15.60:0/1025701861 >> 172.21.15.60:0/1025701861 conn(0x7ff46c0faba0 msgr2=0x7ff46c0fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:26.384 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.405+0000 7ff470ae2700 1 -- 172.21.15.60:0/1025701861 shutdown_connections 2023-07-20T01:27:26.385 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:26.405+0000 7ff470ae2700 1 -- 172.21.15.60:0/1025701861 wait complete. 2023-07-20T01:27:26.385 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 8 2023-07-20T01:27:27.144 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:26 smithi136 ceph-mon[98083]: Deploying daemon mds.h on smithi136 2023-07-20T01:27:27.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:26 smithi136 ceph-mon[98083]: mds.? [v2:172.21.15.183:6836/3540765285,v1:172.21.15.183:6837/3540765285] up:boot 2023-07-20T01:27:27.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:26 smithi136 ceph-mon[98083]: fsmap 7 up:standby 2023-07-20T01:27:27.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:26 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-20T01:27:27.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:26 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/1025701861' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:27.145 INFO:journalctl@ceph.mds.h.smithi136.stdout:Jul 20 01:27:26 smithi136 systemd[1]: Starting Ceph mds.h for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:27:27.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:26 smithi060 ceph-mon[96524]: Deploying daemon mds.h on smithi136 2023-07-20T01:27:27.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:26 smithi060 ceph-mon[96524]: mds.? [v2:172.21.15.183:6836/3540765285,v1:172.21.15.183:6837/3540765285] up:boot 2023-07-20T01:27:27.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:26 smithi060 ceph-mon[96524]: fsmap 7 up:standby 2023-07-20T01:27:27.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:26 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-20T01:27:27.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:26 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/1025701861' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:27.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:26 smithi183 ceph-mon[98477]: Deploying daemon mds.h on smithi136 2023-07-20T01:27:27.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:26 smithi183 ceph-mon[98477]: mds.? [v2:172.21.15.183:6836/3540765285,v1:172.21.15.183:6837/3540765285] up:boot 2023-07-20T01:27:27.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:26 smithi183 ceph-mon[98477]: fsmap 7 up:standby 2023-07-20T01:27:27.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:26 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-20T01:27:27.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:26 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/1025701861' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:27.534 INFO:journalctl@ceph.mds.h.smithi136.stdout:Jul 20 01:27:27 smithi136 podman[122805]: 2023-07-20T01:27:27.894 INFO:journalctl@ceph.mds.h.smithi136.stdout:Jul 20 01:27:27 smithi136 ceph-mds[122845]: -- 172.21.15.136:0/3054213681 <== mon.2 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564de24ed1e0 con 0x564de24c2c00 2023-07-20T01:27:27.894 INFO:journalctl@ceph.mds.h.smithi136.stdout:Jul 20 01:27:27 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-h[122821]: starting mds.h at 2023-07-20T01:27:27.895 INFO:journalctl@ceph.mds.h.smithi136.stdout:Jul 20 01:27:27 smithi136 bash[122805]: 418aedf33069b688901878e06570264cdb6c3f7a72f2e9353542480a51d0499b 2023-07-20T01:27:27.895 INFO:journalctl@ceph.mds.h.smithi136.stdout:Jul 20 01:27:27 smithi136 systemd[1]: Started Ceph mds.h for 7daf031e-269b-11ee-9b34-001a4aab830c. 2023-07-20T01:27:27.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:27 smithi136 ceph-mon[98083]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:27.895 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:27 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:27.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:27 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:27.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:27 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:27.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:27 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:27.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:27 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:27.896 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:27 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:28.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:27 smithi060 ceph-mon[96524]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:28.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:27 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:28.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:27 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:28.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:27 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:28.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:27 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:28.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:27 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:28.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:27 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:28.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:27 smithi183 ceph-mon[98477]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:28.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:27 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:28.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:27 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:28.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:27 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:28.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:27 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:28.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:27 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:28.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:27 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:28.396 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T01:27:28.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.558+0000 7fe2097ce700 1 -- 172.21.15.60:0/1157805201 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe204101210 msgr2=0x7fe204107ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:28.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.558+0000 7fe2097ce700 1 --2- 172.21.15.60:0/1157805201 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe204101210 0x7fe204107ac0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc00b0d0 tx=0x7fe1fc00b3e0 comp rx=0 tx=0).stop 2023-07-20T01:27:28.536 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.558+0000 7fe2097ce700 1 -- 172.21.15.60:0/1157805201 shutdown_connections 2023-07-20T01:27:28.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.558+0000 7fe2097ce700 1 --2- 172.21.15.60:0/1157805201 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe204101210 0x7fe204107ac0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:28.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.558+0000 7fe2097ce700 1 --2- 172.21.15.60:0/1157805201 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe2040ff680 0x7fe2040ffaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:28.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.558+0000 7fe2097ce700 1 --2- 172.21.15.60:0/1157805201 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe204100870 0x7fe204100cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:28.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.558+0000 7fe2097ce700 1 -- 172.21.15.60:0/1157805201 >> 172.21.15.60:0/1157805201 conn(0x7fe2040fabd0 msgr2=0x7fe2040fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:28.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.558+0000 7fe2097ce700 1 -- 172.21.15.60:0/1157805201 shutdown_connections 2023-07-20T01:27:28.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.558+0000 7fe2097ce700 1 -- 172.21.15.60:0/1157805201 wait complete. 2023-07-20T01:27:28.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.558+0000 7fe2097ce700 1 Processor -- start 2023-07-20T01:27:28.537 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.559+0000 7fe2097ce700 1 -- start start 2023-07-20T01:27:28.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.559+0000 7fe2097ce700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe2040ff680 0x7fe204196400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:28.539 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.559+0000 7fe2097ce700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe204100870 0x7fe204196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:28.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.559+0000 7fe2097ce700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe204101210 0x7fe20419d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:28.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.559+0000 7fe2097ce700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe20419df80 con 0x7fe204101210 2023-07-20T01:27:28.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.559+0000 7fe2097ce700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe20419e0c0 con 0x7fe204100870 2023-07-20T01:27:28.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.559+0000 7fe2097ce700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe20419e250 con 0x7fe2040ff680 2023-07-20T01:27:28.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.559+0000 7fe2037fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe204101210 0x7fe20419d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:28.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.559+0000 7fe2037fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe204101210 0x7fe20419d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42954/0 (socket says 172.21.15.60:42954) 2023-07-20T01:27:28.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.559+0000 7fe2037fe700 1 -- 172.21.15.60:0/922300346 learned_addr learned my addr 172.21.15.60:0/922300346 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:27:28.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.559+0000 7fe202ffd700 1 --2- 172.21.15.60:0/922300346 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe2040ff680 0x7fe204196400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:28.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.559+0000 7fe2027fc700 1 --2- 172.21.15.60:0/922300346 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe204100870 0x7fe204196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:28.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.559+0000 7fe2037fe700 1 -- 172.21.15.60:0/922300346 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe2040ff680 msgr2=0x7fe204196400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:28.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.559+0000 7fe2037fe700 1 --2- 172.21.15.60:0/922300346 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe2040ff680 0x7fe204196400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:28.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.559+0000 7fe2037fe700 1 -- 172.21.15.60:0/922300346 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe204100870 msgr2=0x7fe204196940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:28.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.559+0000 7fe2037fe700 1 --2- 172.21.15.60:0/922300346 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe204100870 0x7fe204196940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:28.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.559+0000 7fe2037fe700 1 -- 172.21.15.60:0/922300346 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1fc009d00 con 0x7fe204101210 2023-07-20T01:27:28.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.559+0000 7fe2027fc700 1 --2- 172.21.15.60:0/922300346 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe204100870 0x7fe204196940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T01:27:28.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.559+0000 7fe2037fe700 1 --2- 172.21.15.60:0/922300346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe204101210 0x7fe20419d9b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc007850 tx=0x7fe1fc007880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:28.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.560+0000 7fe1ebfff700 1 -- 172.21.15.60:0/922300346 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc00e040 con 0x7fe204101210 2023-07-20T01:27:28.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.560+0000 7fe2097ce700 1 -- 172.21.15.60:0/922300346 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe20419e4d0 con 0x7fe204101210 2023-07-20T01:27:28.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.560+0000 7fe1ebfff700 1 -- 172.21.15.60:0/922300346 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe1fc003c00 con 0x7fe204101210 2023-07-20T01:27:28.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.560+0000 7fe1ebfff700 1 -- 172.21.15.60:0/922300346 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc01b6c0 con 0x7fe204101210 2023-07-20T01:27:28.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.560+0000 7fe2097ce700 1 -- 172.21.15.60:0/922300346 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe20419e8a0 con 0x7fe204101210 2023-07-20T01:27:28.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.560+0000 7fe2097ce700 1 -- 172.21.15.60:0/922300346 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe20410ad00 con 0x7fe204101210 2023-07-20T01:27:28.543 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.562+0000 7fe1ebfff700 1 -- 172.21.15.60:0/922300346 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe1fc005090 con 0x7fe204101210 2023-07-20T01:27:28.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.562+0000 7fe1ebfff700 1 --2- 172.21.15.60:0/922300346 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fe1ec0a11a0 0x7fe1ec0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:28.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.562+0000 7fe202ffd700 1 --2- 172.21.15.60:0/922300346 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fe1ec0a11a0 0x7fe1ec0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:28.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.562+0000 7fe1ebfff700 1 -- 172.21.15.60:0/922300346 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe1fc0cdba0 con 0x7fe204101210 2023-07-20T01:27:28.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.563+0000 7fe202ffd700 1 --2- 172.21.15.60:0/922300346 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fe1ec0a11a0 0x7fe1ec0a3660 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe2041006d0 tx=0x7fe1f4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:28.544 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.566+0000 7fe1ebfff700 1 -- 172.21.15.60:0/922300346 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe1fc097820 con 0x7fe204101210 2023-07-20T01:27:28.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.812+0000 7fe2097ce700 1 -- 172.21.15.60:0/922300346 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe20419eee0 con 0x7fe204101210 2023-07-20T01:27:28.790 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.813+0000 7fe1ebfff700 1 -- 172.21.15.60:0/922300346 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6568 (secure 0 0 0) 0x7fe1fc09ae40 con 0x7fe204101210 2023-07-20T01:27:28.790 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:27:28.791 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2893530475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2893530475},{"type":"v1","addr":"172.21.15.60:6835","nonce":2893530475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/302154574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":302154574},{"type":"v1","addr":"172.21.15.60:6837","nonce":302154574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/3759595107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3759595107},{"type":"v1","addr":"172.21.15.183:6833","nonce":3759595107}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1706932385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1706932385},{"type":"v1","addr":"172.21.15.136:6833","nonce":1706932385}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3379647260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3379647260},{"type":"v1","addr":"172.21.15.183:6835","nonce":3379647260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3349527335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3349527335},{"type":"v1","addr":"172.21.15.136:6835","nonce":3349527335}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/3540765285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":3540765285},{"type":"v1","addr":"172.21.15.183:6837","nonce":3540765285}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24500,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/895972199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":895972199},{"type":"v1","addr":"172.21.15.136:6837","nonce":895972199}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2023-07-20T01:27:28.792 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.815+0000 7fe2097ce700 1 -- 172.21.15.60:0/922300346 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fe1ec0a11a0 msgr2=0x7fe1ec0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:28.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.815+0000 7fe2097ce700 1 --2- 172.21.15.60:0/922300346 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fe1ec0a11a0 0x7fe1ec0a3660 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe2041006d0 tx=0x7fe1f4009260 comp rx=0 tx=0).stop 2023-07-20T01:27:28.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.815+0000 7fe2097ce700 1 -- 172.21.15.60:0/922300346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe204101210 msgr2=0x7fe20419d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:28.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.815+0000 7fe2097ce700 1 --2- 172.21.15.60:0/922300346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe204101210 0x7fe20419d9b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc007850 tx=0x7fe1fc007880 comp rx=0 tx=0).stop 2023-07-20T01:27:28.793 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.815+0000 7fe2097ce700 1 -- 172.21.15.60:0/922300346 shutdown_connections 2023-07-20T01:27:28.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.815+0000 7fe2097ce700 1 --2- 172.21.15.60:0/922300346 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fe1ec0a11a0 0x7fe1ec0a3660 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:28.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.815+0000 7fe2097ce700 1 --2- 172.21.15.60:0/922300346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe204101210 0x7fe20419d9b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:28.794 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.815+0000 7fe2097ce700 1 --2- 172.21.15.60:0/922300346 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe2040ff680 0x7fe204196400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:28.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.815+0000 7fe2097ce700 1 --2- 172.21.15.60:0/922300346 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe204100870 0x7fe204196940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:28.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.815+0000 7fe2097ce700 1 -- 172.21.15.60:0/922300346 >> 172.21.15.60:0/922300346 conn(0x7fe2040fabd0 msgr2=0x7fe204068bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:28.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.815+0000 7fe2097ce700 1 -- 172.21.15.60:0/922300346 shutdown_connections 2023-07-20T01:27:28.795 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:28.815+0000 7fe2097ce700 1 -- 172.21.15.60:0/922300346 wait complete. 2023-07-20T01:27:28.796 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 9 2023-07-20T01:27:28.926 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:28 smithi183 ceph-mon[98477]: Deploying daemon mds.i on smithi183 2023-07-20T01:27:28.927 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:28 smithi183 ceph-mon[98477]: mds.? [v2:172.21.15.136:6836/895972199,v1:172.21.15.136:6837/895972199] up:boot 2023-07-20T01:27:28.927 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:28 smithi183 ceph-mon[98477]: fsmap 8 up:standby 2023-07-20T01:27:28.927 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:28 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-20T01:27:29.144 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:28 smithi136 ceph-mon[98083]: Deploying daemon mds.i on smithi183 2023-07-20T01:27:29.144 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:28 smithi136 ceph-mon[98083]: mds.? [v2:172.21.15.136:6836/895972199,v1:172.21.15.136:6837/895972199] up:boot 2023-07-20T01:27:29.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:28 smithi136 ceph-mon[98083]: fsmap 8 up:standby 2023-07-20T01:27:29.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:28 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-20T01:27:29.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:28 smithi060 ceph-mon[96524]: Deploying daemon mds.i on smithi183 2023-07-20T01:27:29.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:28 smithi060 ceph-mon[96524]: mds.? [v2:172.21.15.136:6836/895972199,v1:172.21.15.136:6837/895972199] up:boot 2023-07-20T01:27:29.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:28 smithi060 ceph-mon[96524]: fsmap 8 up:standby 2023-07-20T01:27:29.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:28 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-20T01:27:29.447 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 20 01:27:29 smithi183 systemd[1]: Starting Ceph mds.i for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:27:29.725 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 20 01:27:29 smithi183 podman[124511]: 2023-07-20T01:27:30.144 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:29 smithi136 ceph-mon[98083]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:30.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:29 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/922300346' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:30.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:29 smithi060 ceph-mon[96524]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:30.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:29 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/922300346' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:30.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:29 smithi183 ceph-mon[98477]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:30.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:29 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/922300346' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:30.197 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 20 01:27:29 smithi183 ceph-mds[124552]: -- 172.21.15.183:0/122354932 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555fc72d71e0 con 0x555fc72ac800 2023-07-20T01:27:30.198 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 20 01:27:29 smithi183 bash[124511]: 90298e59be754f5032a8bde066363cbd3936d036b76ece9aebea52b9ab80a31b 2023-07-20T01:27:30.198 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 20 01:27:29 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-i[124527]: starting mds.i at 2023-07-20T01:27:30.198 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 20 01:27:29 smithi183 systemd[1]: Started Ceph mds.i for 7daf031e-269b-11ee-9b34-001a4aab830c. 2023-07-20T01:27:30.805 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T01:27:30.834 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:30 smithi060 ceph-mon[96524]: mds.? [v2:172.21.15.183:6838/2229055327,v1:172.21.15.183:6839/2229055327] up:boot 2023-07-20T01:27:30.835 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:30 smithi060 ceph-mon[96524]: fsmap 9 up:standby 2023-07-20T01:27:30.835 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:30 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-20T01:27:30.835 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:30 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:30.836 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:30 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:30.836 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:30 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:30.836 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:30 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:30.837 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:30 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:30.837 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:30 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:30.939 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.961+0000 7f848b04a700 1 -- 172.21.15.60:0/2619600686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8484100840 msgr2=0x7f8484100ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:30.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.961+0000 7f848b04a700 1 --2- 172.21.15.60:0/2619600686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8484100840 0x7f8484100ca0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f84700099c0 tx=0x7f8470009cd0 comp rx=0 tx=0).stop 2023-07-20T01:27:30.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.961+0000 7f848b04a700 1 -- 172.21.15.60:0/2619600686 shutdown_connections 2023-07-20T01:27:30.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.961+0000 7f848b04a700 1 --2- 172.21.15.60:0/2619600686 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84841011e0 0x7f8484107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:30.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.961+0000 7f848b04a700 1 --2- 172.21.15.60:0/2619600686 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84840ff650 0x7f84840ffa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:30.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.961+0000 7f848b04a700 1 --2- 172.21.15.60:0/2619600686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8484100840 0x7f8484100ca0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:30.940 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.961+0000 7f848b04a700 1 -- 172.21.15.60:0/2619600686 >> 172.21.15.60:0/2619600686 conn(0x7f84840faba0 msgr2=0x7f84840fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:30.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.961+0000 7f848b04a700 1 -- 172.21.15.60:0/2619600686 shutdown_connections 2023-07-20T01:27:30.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.961+0000 7f848b04a700 1 -- 172.21.15.60:0/2619600686 wait complete. 2023-07-20T01:27:30.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.962+0000 7f848b04a700 1 Processor -- start 2023-07-20T01:27:30.941 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.962+0000 7f848b04a700 1 -- start start 2023-07-20T01:27:30.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.962+0000 7f848b04a700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84840ff650 0x7f84841035e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:30.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.962+0000 7f848b04a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8484100840 0x7f8484101c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:30.943 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.962+0000 7f848b04a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84841011e0 0x7f8484102160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:30.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.962+0000 7f848b04a700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8484102900 con 0x7f8484100840 2023-07-20T01:27:30.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.962+0000 7f848b04a700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8484102a70 con 0x7f84841011e0 2023-07-20T01:27:30.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.962+0000 7f848b04a700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84841a03c0 con 0x7f84840ff650 2023-07-20T01:27:30.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.962+0000 7f8483fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8484100840 0x7f8484101c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:30.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.962+0000 7f8483fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8484100840 0x7f8484101c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42964/0 (socket says 172.21.15.60:42964) 2023-07-20T01:27:30.944 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.962+0000 7f8483fff700 1 -- 172.21.15.60:0/71468761 learned_addr learned my addr 172.21.15.60:0/71468761 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:27:30.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.962+0000 7f8488de6700 1 --2- 172.21.15.60:0/71468761 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84840ff650 0x7f84841035e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:30.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.962+0000 7f84895e7700 1 --2- 172.21.15.60:0/71468761 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84841011e0 0x7f8484102160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:30.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.963+0000 7f8483fff700 1 -- 172.21.15.60:0/71468761 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84840ff650 msgr2=0x7f84841035e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:30.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.963+0000 7f8483fff700 1 --2- 172.21.15.60:0/71468761 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84840ff650 0x7f84841035e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:30.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.963+0000 7f8483fff700 1 -- 172.21.15.60:0/71468761 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84841011e0 msgr2=0x7f8484102160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:30.945 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.963+0000 7f8483fff700 1 --2- 172.21.15.60:0/71468761 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84841011e0 0x7f8484102160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:30.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.963+0000 7f8483fff700 1 -- 172.21.15.60:0/71468761 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84700095f0 con 0x7f8484100840 2023-07-20T01:27:30.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.963+0000 7f8483fff700 1 --2- 172.21.15.60:0/71468761 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8484100840 0x7f8484101c20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8470005910 tx=0x7f8470003a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:30.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.963+0000 7f8481ffb700 1 -- 172.21.15.60:0/71468761 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847001d070 con 0x7f8484100840 2023-07-20T01:27:30.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.963+0000 7f8481ffb700 1 -- 172.21.15.60:0/71468761 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f847000bd30 con 0x7f8484100840 2023-07-20T01:27:30.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.963+0000 7f8481ffb700 1 -- 172.21.15.60:0/71468761 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8470022ca0 con 0x7f8484100840 2023-07-20T01:27:30.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.963+0000 7f848b04a700 1 -- 172.21.15.60:0/71468761 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84841a0640 con 0x7f8484100840 2023-07-20T01:27:30.946 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.963+0000 7f848b04a700 1 -- 172.21.15.60:0/71468761 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84841a0a10 con 0x7f8484100840 2023-07-20T01:27:30.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.964+0000 7f848b04a700 1 -- 172.21.15.60:0/71468761 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f848410ad10 con 0x7f8484100840 2023-07-20T01:27:30.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.965+0000 7f8481ffb700 1 -- 172.21.15.60:0/71468761 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8470022470 con 0x7f8484100840 2023-07-20T01:27:30.947 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.966+0000 7f8481ffb700 1 --2- 172.21.15.60:0/71468761 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f84740a1220 0x7f84740a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:30.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.966+0000 7f8488de6700 1 --2- 172.21.15.60:0/71468761 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f84740a1220 0x7f84740a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:30.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.966+0000 7f8481ffb700 1 -- 172.21.15.60:0/71468761 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f84700ce180 con 0x7f8484100840 2023-07-20T01:27:30.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.966+0000 7f8488de6700 1 --2- 172.21.15.60:0/71468761 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f84740a1220 0x7f84740a36e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f84841006a0 tx=0x7f847800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:30.948 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:30.969+0000 7f8481ffb700 1 -- 172.21.15.60:0/71468761 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8470097d60 con 0x7f8484100840 2023-07-20T01:27:31.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:30 smithi136 ceph-mon[98083]: mds.? [v2:172.21.15.183:6838/2229055327,v1:172.21.15.183:6839/2229055327] up:boot 2023-07-20T01:27:31.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:30 smithi136 ceph-mon[98083]: fsmap 9 up:standby 2023-07-20T01:27:31.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:30 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-20T01:27:31.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:30 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:31.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:30 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:31.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:30 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:31.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:30 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:31.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:30 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:31.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:30 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:31.192 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:31.214+0000 7f848b04a700 1 -- 172.21.15.60:0/71468761 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84841a1050 con 0x7f8484100840 2023-07-20T01:27:31.193 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:31.215+0000 7f8481ffb700 1 -- 172.21.15.60:0/71468761 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7330 (secure 0 0 0) 0x7f847009b380 con 0x7f8484100840 2023-07-20T01:27:31.193 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:27:31.194 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2893530475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2893530475},{"type":"v1","addr":"172.21.15.60:6835","nonce":2893530475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/302154574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":302154574},{"type":"v1","addr":"172.21.15.60:6837","nonce":302154574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/3759595107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3759595107},{"type":"v1","addr":"172.21.15.183:6833","nonce":3759595107}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1706932385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1706932385},{"type":"v1","addr":"172.21.15.136:6833","nonce":1706932385}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3379647260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3379647260},{"type":"v1","addr":"172.21.15.183:6835","nonce":3379647260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3349527335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3349527335},{"type":"v1","addr":"172.21.15.136:6835","nonce":3349527335}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/3540765285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":3540765285},{"type":"v1","addr":"172.21.15.183:6837","nonce":3540765285}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/2229055327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":2229055327},{"type":"v1","addr":"172.21.15.183:6839","nonce":2229055327}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24500,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/895972199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":895972199},{"type":"v1","addr":"172.21.15.136:6837","nonce":895972199}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2023-07-20T01:27:31.195 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:31.217+0000 7f848b04a700 1 -- 172.21.15.60:0/71468761 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f84740a1220 msgr2=0x7f84740a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:31.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:31.217+0000 7f848b04a700 1 --2- 172.21.15.60:0/71468761 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f84740a1220 0x7f84740a36e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f84841006a0 tx=0x7f847800b410 comp rx=0 tx=0).stop 2023-07-20T01:27:31.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:31.217+0000 7f848b04a700 1 -- 172.21.15.60:0/71468761 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8484100840 msgr2=0x7f8484101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:31.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:31.217+0000 7f848b04a700 1 --2- 172.21.15.60:0/71468761 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8484100840 0x7f8484101c20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8470005910 tx=0x7f8470003a50 comp rx=0 tx=0).stop 2023-07-20T01:27:31.196 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:31.217+0000 7f848b04a700 1 -- 172.21.15.60:0/71468761 shutdown_connections 2023-07-20T01:27:31.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:31.217+0000 7f848b04a700 1 --2- 172.21.15.60:0/71468761 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f84740a1220 0x7f84740a36e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:31.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:31.217+0000 7f848b04a700 1 --2- 172.21.15.60:0/71468761 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84841011e0 0x7f8484102160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:31.197 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:31.217+0000 7f848b04a700 1 --2- 172.21.15.60:0/71468761 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f84840ff650 0x7f84841035e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:31.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:31.217+0000 7f848b04a700 1 --2- 172.21.15.60:0/71468761 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8484100840 0x7f8484101c20 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:31.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:31.217+0000 7f848b04a700 1 -- 172.21.15.60:0/71468761 >> 172.21.15.60:0/71468761 conn(0x7f84840faba0 msgr2=0x7f84840678f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:31.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:31.217+0000 7f848b04a700 1 -- 172.21.15.60:0/71468761 shutdown_connections 2023-07-20T01:27:31.198 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:31.217+0000 7f848b04a700 1 -- 172.21.15.60:0/71468761 wait complete. 2023-07-20T01:27:31.198 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 10 2023-07-20T01:27:31.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:30 smithi183 ceph-mon[98477]: mds.? [v2:172.21.15.183:6838/2229055327,v1:172.21.15.183:6839/2229055327] up:boot 2023-07-20T01:27:31.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:30 smithi183 ceph-mon[98477]: fsmap 9 up:standby 2023-07-20T01:27:31.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:30 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-20T01:27:31.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:30 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:31.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:30 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:31.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:30 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:31.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:30 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:31.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:30 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:31.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:30 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:31.674 INFO:journalctl@ceph.mds.g.smithi060.stdout:Jul 20 01:27:31 smithi060 systemd[1]: Starting Ceph mds.g for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:27:31.998 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:31 smithi060 ceph-mon[96524]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:31.999 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:31 smithi060 ceph-mon[96524]: Deploying daemon mds.g on smithi060 2023-07-20T01:27:31.999 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:31 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/71468761' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:31.999 INFO:journalctl@ceph.mds.g.smithi060.stdout:Jul 20 01:27:31 smithi060 podman[130717]: 2023-07-20T01:27:32.144 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:31 smithi136 ceph-mon[98083]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:32.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:31 smithi136 ceph-mon[98083]: Deploying daemon mds.g on smithi060 2023-07-20T01:27:32.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:31 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/71468761' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:32.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:31 smithi183 ceph-mon[98477]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:32.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:31 smithi183 ceph-mon[98477]: Deploying daemon mds.g on smithi060 2023-07-20T01:27:32.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:31 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/71468761' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:32.424 INFO:journalctl@ceph.mds.g.smithi060.stdout:Jul 20 01:27:32 smithi060 ceph-mds[130757]: -- 172.21.15.60:0/2433480581 <== mon.0 v2:172.21.15.60:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564e587631e0 con 0x564e58739000 2023-07-20T01:27:32.425 INFO:journalctl@ceph.mds.g.smithi060.stdout:Jul 20 01:27:32 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-g[130733]: starting mds.g at 2023-07-20T01:27:32.425 INFO:journalctl@ceph.mds.g.smithi060.stdout:Jul 20 01:27:32 smithi060 bash[130717]: 0637eebfb4908882feb319086aefa522f25a969ad499eecf855cf4f584b30b89 2023-07-20T01:27:32.425 INFO:journalctl@ceph.mds.g.smithi060.stdout:Jul 20 01:27:32 smithi060 systemd[1]: Started Ceph mds.g for 7daf031e-269b-11ee-9b34-001a4aab830c. 2023-07-20T01:27:33.207 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T01:27:33.235 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:33 smithi060 ceph-mon[96524]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:33.235 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:33 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:33.235 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:33 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:33.235 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:33 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:33.235 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:33 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:33.236 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:33 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:33.236 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:33 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:33.236 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:33 smithi060 ceph-mon[96524]: Deploying daemon mds.a on smithi060 2023-07-20T01:27:33.236 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:33 smithi060 ceph-mon[96524]: mds.? [v2:172.21.15.60:6838/3584660034,v1:172.21.15.60:6839/3584660034] up:boot 2023-07-20T01:27:33.236 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:33 smithi060 ceph-mon[96524]: fsmap 10 up:standby 2023-07-20T01:27:33.237 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:33 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-20T01:27:33.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.361+0000 7f3144825700 1 -- 172.21.15.60:0/4158342111 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31401011e0 msgr2=0x7f3140107a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:33.339 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.361+0000 7f3144825700 1 --2- 172.21.15.60:0/4158342111 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31401011e0 0x7f3140107a90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f313800b0d0 tx=0x7f313800b3e0 comp rx=0 tx=0).stop 2023-07-20T01:27:33.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.361+0000 7f3144825700 1 -- 172.21.15.60:0/4158342111 shutdown_connections 2023-07-20T01:27:33.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.361+0000 7f3144825700 1 --2- 172.21.15.60:0/4158342111 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31401011e0 0x7f3140107a90 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:33.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.361+0000 7f3144825700 1 --2- 172.21.15.60:0/4158342111 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31400ff650 0x7f31400ffa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:33.340 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.361+0000 7f3144825700 1 --2- 172.21.15.60:0/4158342111 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3140100840 0x7f3140100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:33.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.361+0000 7f3144825700 1 -- 172.21.15.60:0/4158342111 >> 172.21.15.60:0/4158342111 conn(0x7f31400faba0 msgr2=0x7f31400fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:33.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.362+0000 7f3144825700 1 -- 172.21.15.60:0/4158342111 shutdown_connections 2023-07-20T01:27:33.341 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.362+0000 7f3144825700 1 -- 172.21.15.60:0/4158342111 wait complete. 2023-07-20T01:27:33.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.363+0000 7f3144825700 1 Processor -- start 2023-07-20T01:27:33.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.363+0000 7f3144825700 1 -- start start 2023-07-20T01:27:33.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.363+0000 7f3144825700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31400ff650 0x7f3140068440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:33.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.363+0000 7f3144825700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3140100840 0x7f314019d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:33.343 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.363+0000 7f3144825700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31401011e0 0x7f314019fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:33.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.363+0000 7f3144825700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31401a01d0 con 0x7f31401011e0 2023-07-20T01:27:33.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.363+0000 7f3144825700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31401a0340 con 0x7f3140100840 2023-07-20T01:27:33.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.363+0000 7f3144825700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31401a0500 con 0x7f31400ff650 2023-07-20T01:27:33.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.363+0000 7f313ed9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31401011e0 0x7f314019fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:33.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.363+0000 7f313ed9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31401011e0 0x7f314019fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42994/0 (socket says 172.21.15.60:42994) 2023-07-20T01:27:33.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.363+0000 7f313ed9d700 1 -- 172.21.15.60:0/3343435204 learned_addr learned my addr 172.21.15.60:0/3343435204 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:27:33.344 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.363+0000 7f313e59c700 1 --2- 172.21.15.60:0/3343435204 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31400ff650 0x7f3140068440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:33.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.363+0000 7f313ed9d700 1 -- 172.21.15.60:0/3343435204 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31400ff650 msgr2=0x7f3140068440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:33.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.363+0000 7f313ed9d700 1 --2- 172.21.15.60:0/3343435204 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31400ff650 0x7f3140068440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:33.345 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.363+0000 7f313dd9b700 1 --2- 172.21.15.60:0/3343435204 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3140100840 0x7f314019d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:33.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.363+0000 7f313ed9d700 1 -- 172.21.15.60:0/3343435204 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3140100840 msgr2=0x7f314019d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:33.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.363+0000 7f313ed9d700 1 --2- 172.21.15.60:0/3343435204 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3140100840 0x7f314019d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:33.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.363+0000 7f313ed9d700 1 -- 172.21.15.60:0/3343435204 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3138009d00 con 0x7f31401011e0 2023-07-20T01:27:33.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.363+0000 7f313dd9b700 1 --2- 172.21.15.60:0/3343435204 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3140100840 0x7f314019d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T01:27:33.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.363+0000 7f313ed9d700 1 --2- 172.21.15.60:0/3343435204 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31401011e0 0x7f314019fb20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3138009440 tx=0x7f3138009470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:33.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.364+0000 7f312f7fe700 1 -- 172.21.15.60:0/3343435204 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313800e040 con 0x7f31401011e0 2023-07-20T01:27:33.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.364+0000 7f3144825700 1 -- 172.21.15.60:0/3343435204 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31401a0780 con 0x7f31401011e0 2023-07-20T01:27:33.346 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.364+0000 7f312f7fe700 1 -- 172.21.15.60:0/3343435204 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f3138012e10 con 0x7f31401011e0 2023-07-20T01:27:33.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.364+0000 7f312f7fe700 1 -- 172.21.15.60:0/3343435204 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313801b450 con 0x7f31401011e0 2023-07-20T01:27:33.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.364+0000 7f3144825700 1 -- 172.21.15.60:0/3343435204 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31401a0c00 con 0x7f31401011e0 2023-07-20T01:27:33.347 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.365+0000 7f3144825700 1 -- 172.21.15.60:0/3343435204 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f314010ad70 con 0x7f31401011e0 2023-07-20T01:27:33.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.366+0000 7f312f7fe700 1 -- 172.21.15.60:0/3343435204 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3138012980 con 0x7f31401011e0 2023-07-20T01:27:33.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.366+0000 7f312f7fe700 1 --2- 172.21.15.60:0/3343435204 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f31280a11a0 0x7f31280a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:33.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.367+0000 7f313e59c700 1 --2- 172.21.15.60:0/3343435204 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f31280a11a0 0x7f31280a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:33.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.367+0000 7f312f7fe700 1 -- 172.21.15.60:0/3343435204 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f31380ce780 con 0x7f31401011e0 2023-07-20T01:27:33.348 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.367+0000 7f313e59c700 1 --2- 172.21.15.60:0/3343435204 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f31280a11a0 0x7f31280a3660 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f31401006a0 tx=0x7f313400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:33.349 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.370+0000 7f312f7fe700 1 -- 172.21.15.60:0/3343435204 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3138098400 con 0x7f31401011e0 2023-07-20T01:27:33.394 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:33 smithi136 ceph-mon[98083]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:33.394 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:33 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:33.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:33 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:33.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:33 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:33.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:33 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:33.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:33 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:33.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:33 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:33.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:33 smithi136 ceph-mon[98083]: Deploying daemon mds.a on smithi060 2023-07-20T01:27:33.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:33 smithi136 ceph-mon[98083]: mds.? [v2:172.21.15.60:6838/3584660034,v1:172.21.15.60:6839/3584660034] up:boot 2023-07-20T01:27:33.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:33 smithi136 ceph-mon[98083]: fsmap 10 up:standby 2023-07-20T01:27:33.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:33 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-20T01:27:33.447 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:33 smithi183 ceph-mon[98477]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:33.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:33 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:33.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:33 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:33.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:33 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:33.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:33 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:33.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:33 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:33.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:33 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:33.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:33 smithi183 ceph-mon[98477]: Deploying daemon mds.a on smithi060 2023-07-20T01:27:33.449 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:33 smithi183 ceph-mon[98477]: mds.? [v2:172.21.15.60:6838/3584660034,v1:172.21.15.60:6839/3584660034] up:boot 2023-07-20T01:27:33.449 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:33 smithi183 ceph-mon[98477]: fsmap 10 up:standby 2023-07-20T01:27:33.449 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:33 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-20T01:27:33.594 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.617+0000 7f3144825700 1 -- 172.21.15.60:0/3343435204 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3140106540 con 0x7f31401011e0 2023-07-20T01:27:33.596 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.618+0000 7f312f7fe700 1 -- 172.21.15.60:0/3343435204 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8088 (secure 0 0 0) 0x7f3138011390 con 0x7f31401011e0 2023-07-20T01:27:33.596 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:27:33.596 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2893530475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2893530475},{"type":"v1","addr":"172.21.15.60:6835","nonce":2893530475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/302154574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":302154574},{"type":"v1","addr":"172.21.15.60:6837","nonce":302154574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6839/3584660034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":3584660034},{"type":"v1","addr":"172.21.15.60:6839","nonce":3584660034}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/3759595107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3759595107},{"type":"v1","addr":"172.21.15.183:6833","nonce":3759595107}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1706932385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1706932385},{"type":"v1","addr":"172.21.15.136:6833","nonce":1706932385}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3379647260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3379647260},{"type":"v1","addr":"172.21.15.183:6835","nonce":3379647260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3349527335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3349527335},{"type":"v1","addr":"172.21.15.136:6835","nonce":3349527335}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/3540765285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":3540765285},{"type":"v1","addr":"172.21.15.183:6837","nonce":3540765285}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/2229055327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":2229055327},{"type":"v1","addr":"172.21.15.183:6839","nonce":2229055327}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24500,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/895972199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":895972199},{"type":"v1","addr":"172.21.15.136:6837","nonce":895972199}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2023-07-20T01:27:33.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.620+0000 7f3144825700 1 -- 172.21.15.60:0/3343435204 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f31280a11a0 msgr2=0x7f31280a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:33.598 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.620+0000 7f3144825700 1 --2- 172.21.15.60:0/3343435204 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f31280a11a0 0x7f31280a3660 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f31401006a0 tx=0x7f313400b410 comp rx=0 tx=0).stop 2023-07-20T01:27:33.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.620+0000 7f3144825700 1 -- 172.21.15.60:0/3343435204 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31401011e0 msgr2=0x7f314019fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:33.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.620+0000 7f3144825700 1 --2- 172.21.15.60:0/3343435204 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31401011e0 0x7f314019fb20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3138009440 tx=0x7f3138009470 comp rx=0 tx=0).stop 2023-07-20T01:27:33.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.620+0000 7f3144825700 1 -- 172.21.15.60:0/3343435204 shutdown_connections 2023-07-20T01:27:33.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.620+0000 7f3144825700 1 --2- 172.21.15.60:0/3343435204 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f31280a11a0 0x7f31280a3660 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:33.599 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.620+0000 7f3144825700 1 --2- 172.21.15.60:0/3343435204 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31401011e0 0x7f314019fb20 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:33.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.620+0000 7f3144825700 1 --2- 172.21.15.60:0/3343435204 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31400ff650 0x7f3140068440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:33.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.620+0000 7f3144825700 1 --2- 172.21.15.60:0/3343435204 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3140100840 0x7f314019d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:33.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.620+0000 7f3144825700 1 -- 172.21.15.60:0/3343435204 >> 172.21.15.60:0/3343435204 conn(0x7f31400faba0 msgr2=0x7f31400fce30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:33.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.621+0000 7f3144825700 1 -- 172.21.15.60:0/3343435204 shutdown_connections 2023-07-20T01:27:33.600 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:33.621+0000 7f3144825700 1 -- 172.21.15.60:0/3343435204 wait complete. 2023-07-20T01:27:33.601 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 11 2023-07-20T01:27:33.841 INFO:journalctl@ceph.mds.a.smithi060.stdout:Jul 20 01:27:33 smithi060 systemd[1]: Starting Ceph mds.a for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:27:34.390 INFO:journalctl@ceph.mds.a.smithi060.stdout:Jul 20 01:27:34 smithi060 podman[131102]: 2023-07-20T01:27:34.390 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:34 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/3343435204' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:34.394 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:34 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/3343435204' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:34.447 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:34 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/3343435204' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:34.674 INFO:journalctl@ceph.mds.a.smithi060.stdout:Jul 20 01:27:34 smithi060 ceph-mds[131142]: -- 172.21.15.60:0/1269208515 <== mon.0 v2:172.21.15.60:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5632f71a11e0 con 0x5632f7176800 2023-07-20T01:27:34.674 INFO:journalctl@ceph.mds.a.smithi060.stdout:Jul 20 01:27:34 smithi060 bash[131102]: e8911f18b6681e2c0544cb84a12fabc2de5e7ac71fd8c75b72586674accd1a97 2023-07-20T01:27:34.674 INFO:journalctl@ceph.mds.a.smithi060.stdout:Jul 20 01:27:34 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-a[131118]: starting mds.a at 2023-07-20T01:27:34.674 INFO:journalctl@ceph.mds.a.smithi060.stdout:Jul 20 01:27:34 smithi060 systemd[1]: Started Ceph mds.a for 7daf031e-269b-11ee-9b34-001a4aab830c. 2023-07-20T01:27:35.480 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:35 smithi136 ceph-mon[98083]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:35.481 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:35 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:35.481 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:35 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:35.482 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:35 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:35.482 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:35 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:35.482 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:35 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:35.483 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:35 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:35.483 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:35 smithi136 ceph-mon[98083]: Deploying daemon mds.b on smithi136 2023-07-20T01:27:35.483 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:35 smithi136 ceph-mon[98083]: mds.? [v2:172.21.15.60:6840/2905977980,v1:172.21.15.60:6841/2905977980] up:boot 2023-07-20T01:27:35.483 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:35 smithi136 ceph-mon[98083]: fsmap 11 up:standby 2023-07-20T01:27:35.483 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:35 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-20T01:27:35.620 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T01:27:35.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.788+0000 7fb07e47f700 1 -- 172.21.15.60:0/800750773 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb078100220 msgr2=0x7fb078107ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:35.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.788+0000 7fb07e47f700 1 --2- 172.21.15.60:0/800750773 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb078100220 0x7fb078107ac0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb07000b0a0 tx=0x7fb07000b3b0 comp rx=0 tx=0).stop 2023-07-20T01:27:35.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.788+0000 7fb07e47f700 1 -- 172.21.15.60:0/800750773 shutdown_connections 2023-07-20T01:27:35.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.788+0000 7fb07e47f700 1 --2- 172.21.15.60:0/800750773 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb078100220 0x7fb078107ac0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:35.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.788+0000 7fb07e47f700 1 --2- 172.21.15.60:0/800750773 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb078100c10 0x7fb078101030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:35.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.788+0000 7fb07e47f700 1 --2- 172.21.15.60:0/800750773 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb0780ff860 0x7fb0780ffce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:35.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.788+0000 7fb07e47f700 1 -- 172.21.15.60:0/800750773 >> 172.21.15.60:0/800750773 conn(0x7fb0780fa330 msgr2=0x7fb0780fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:35.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.788+0000 7fb07e47f700 1 -- 172.21.15.60:0/800750773 shutdown_connections 2023-07-20T01:27:35.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.788+0000 7fb07e47f700 1 -- 172.21.15.60:0/800750773 wait complete. 2023-07-20T01:27:35.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.789+0000 7fb07e47f700 1 Processor -- start 2023-07-20T01:27:35.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.789+0000 7fb07e47f700 1 -- start start 2023-07-20T01:27:35.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.789+0000 7fb07e47f700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0780ff860 0x7fb0781964a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:35.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.789+0000 7fb07e47f700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb078100220 0x7fb0781969e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:35.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.789+0000 7fb07e47f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb078100c10 0x7fb07819da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:35.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.789+0000 7fb07e47f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07819e020 con 0x7fb078100c10 2023-07-20T01:27:35.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.789+0000 7fb07e47f700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07819e160 con 0x7fb078100220 2023-07-20T01:27:35.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.789+0000 7fb07e47f700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07819e2f0 con 0x7fb0780ff860 2023-07-20T01:27:35.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.789+0000 7fb07ca1c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb078100c10 0x7fb07819da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:35.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.789+0000 7fb07ca1c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb078100c10 0x7fb07819da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43020/0 (socket says 172.21.15.60:43020) 2023-07-20T01:27:35.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.789+0000 7fb07ca1c700 1 -- 172.21.15.60:0/2308505391 learned_addr learned my addr 172.21.15.60:0/2308505391 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:27:35.770 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.789+0000 7fb077fff700 1 --2- 172.21.15.60:0/2308505391 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0780ff860 0x7fb0781964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:35.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.789+0000 7fb06ffff700 1 --2- 172.21.15.60:0/2308505391 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb078100220 0x7fb0781969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:35.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.789+0000 7fb07ca1c700 1 -- 172.21.15.60:0/2308505391 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0780ff860 msgr2=0x7fb0781964a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:35.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.789+0000 7fb07ca1c700 1 --2- 172.21.15.60:0/2308505391 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0780ff860 0x7fb0781964a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:35.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.789+0000 7fb07ca1c700 1 -- 172.21.15.60:0/2308505391 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb078100220 msgr2=0x7fb0781969e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:35.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.789+0000 7fb07ca1c700 1 --2- 172.21.15.60:0/2308505391 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb078100220 0x7fb0781969e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:35.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.789+0000 7fb07ca1c700 1 -- 172.21.15.60:0/2308505391 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb070009d00 con 0x7fb078100c10 2023-07-20T01:27:35.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.789+0000 7fb06ffff700 1 --2- 172.21.15.60:0/2308505391 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb078100220 0x7fb0781969e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T01:27:35.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.789+0000 7fb07ca1c700 1 --2- 172.21.15.60:0/2308505391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb078100c10 0x7fb07819da50 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb07000b070 tx=0x7fb070004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:35.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.789+0000 7fb075ffb700 1 -- 172.21.15.60:0/2308505391 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07000e040 con 0x7fb078100c10 2023-07-20T01:27:35.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.790+0000 7fb075ffb700 1 -- 172.21.15.60:0/2308505391 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb070003e90 con 0x7fb078100c10 2023-07-20T01:27:35.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.790+0000 7fb07e47f700 1 -- 172.21.15.60:0/2308505391 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb07819e570 con 0x7fb078100c10 2023-07-20T01:27:35.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.790+0000 7fb075ffb700 1 -- 172.21.15.60:0/2308505391 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb070004020 con 0x7fb078100c10 2023-07-20T01:27:35.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.790+0000 7fb07e47f700 1 -- 172.21.15.60:0/2308505391 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb07819e9f0 con 0x7fb078100c10 2023-07-20T01:27:35.774 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.791+0000 7fb07e47f700 1 -- 172.21.15.60:0/2308505391 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb07810ad50 con 0x7fb078100c10 2023-07-20T01:27:35.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.792+0000 7fb075ffb700 1 -- 172.21.15.60:0/2308505391 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb070003a00 con 0x7fb078100c10 2023-07-20T01:27:35.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.792+0000 7fb075ffb700 1 --2- 172.21.15.60:0/2308505391 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fb0640a11a0 0x7fb0640a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:35.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.793+0000 7fb077fff700 1 --2- 172.21.15.60:0/2308505391 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fb0640a11a0 0x7fb0640a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:35.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.793+0000 7fb075ffb700 1 -- 172.21.15.60:0/2308505391 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb0700ce220 con 0x7fb078100c10 2023-07-20T01:27:35.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.793+0000 7fb077fff700 1 --2- 172.21.15.60:0/2308505391 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fb0640a11a0 0x7fb0640a3660 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb068005950 tx=0x7fb06800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:35.775 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:35.796+0000 7fb075ffb700 1 -- 172.21.15.60:0/2308505391 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb070097e00 con 0x7fb078100c10 2023-07-20T01:27:35.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:35 smithi060 ceph-mon[96524]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:35.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:35 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:35.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:35 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:35.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:35 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:35.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:35 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:35.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:35 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:35.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:35 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:35.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:35 smithi060 ceph-mon[96524]: Deploying daemon mds.b on smithi136 2023-07-20T01:27:35.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:35 smithi060 ceph-mon[96524]: mds.? [v2:172.21.15.60:6840/2905977980,v1:172.21.15.60:6841/2905977980] up:boot 2023-07-20T01:27:35.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:35 smithi060 ceph-mon[96524]: fsmap 11 up:standby 2023-07-20T01:27:35.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:35 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-20T01:27:35.947 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:35 smithi183 ceph-mon[98477]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:35.947 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:35 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:35.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:35 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:35.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:35 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:35.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:35 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-20T01:27:35.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:35 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-20T01:27:35.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:35 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:35.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:35 smithi183 ceph-mon[98477]: Deploying daemon mds.b on smithi136 2023-07-20T01:27:35.948 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:35 smithi183 ceph-mon[98477]: mds.? [v2:172.21.15.60:6840/2905977980,v1:172.21.15.60:6841/2905977980] up:boot 2023-07-20T01:27:35.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:35 smithi183 ceph-mon[98477]: fsmap 11 up:standby 2023-07-20T01:27:35.949 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:35 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-20T01:27:36.017 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:36.039+0000 7fb07e47f700 1 -- 172.21.15.60:0/2308505391 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb07819f030 con 0x7fb078100c10 2023-07-20T01:27:36.018 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:36.040+0000 7fb075ffb700 1 -- 172.21.15.60:0/2308505391 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+8846 (secure 0 0 0) 0x7fb070011390 con 0x7fb078100c10 2023-07-20T01:27:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:27:36.018 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2893530475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2893530475},{"type":"v1","addr":"172.21.15.60:6835","nonce":2893530475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/302154574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":302154574},{"type":"v1","addr":"172.21.15.60:6837","nonce":302154574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6839/3584660034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":3584660034},{"type":"v1","addr":"172.21.15.60:6839","nonce":3584660034}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6841/2905977980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6840","nonce":2905977980},{"type":"v1","addr":"172.21.15.60:6841","nonce":2905977980}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/3759595107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3759595107},{"type":"v1","addr":"172.21.15.183:6833","nonce":3759595107}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1706932385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1706932385},{"type":"v1","addr":"172.21.15.136:6833","nonce":1706932385}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3379647260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3379647260},{"type":"v1","addr":"172.21.15.183:6835","nonce":3379647260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3349527335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3349527335},{"type":"v1","addr":"172.21.15.136:6835","nonce":3349527335}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/3540765285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":3540765285},{"type":"v1","addr":"172.21.15.183:6837","nonce":3540765285}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/2229055327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":2229055327},{"type":"v1","addr":"172.21.15.183:6839","nonce":2229055327}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24500,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/895972199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":895972199},{"type":"v1","addr":"172.21.15.136:6837","nonce":895972199}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2023-07-20T01:27:36.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:36.042+0000 7fb07e47f700 1 -- 172.21.15.60:0/2308505391 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fb0640a11a0 msgr2=0x7fb0640a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:36.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:36.042+0000 7fb07e47f700 1 --2- 172.21.15.60:0/2308505391 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fb0640a11a0 0x7fb0640a3660 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb068005950 tx=0x7fb06800b410 comp rx=0 tx=0).stop 2023-07-20T01:27:36.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:36.042+0000 7fb07e47f700 1 -- 172.21.15.60:0/2308505391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb078100c10 msgr2=0x7fb07819da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:36.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:36.042+0000 7fb07e47f700 1 --2- 172.21.15.60:0/2308505391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb078100c10 0x7fb07819da50 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb07000b070 tx=0x7fb070004c80 comp rx=0 tx=0).stop 2023-07-20T01:27:36.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:36.042+0000 7fb07e47f700 1 -- 172.21.15.60:0/2308505391 shutdown_connections 2023-07-20T01:27:36.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:36.042+0000 7fb07e47f700 1 --2- 172.21.15.60:0/2308505391 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fb0640a11a0 0x7fb0640a3660 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:36.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:36.042+0000 7fb07e47f700 1 --2- 172.21.15.60:0/2308505391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb078100c10 0x7fb07819da50 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:36.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:36.042+0000 7fb07e47f700 1 --2- 172.21.15.60:0/2308505391 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0780ff860 0x7fb0781964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:36.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:36.042+0000 7fb07e47f700 1 --2- 172.21.15.60:0/2308505391 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb078100220 0x7fb0781969e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:36.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:36.042+0000 7fb07e47f700 1 -- 172.21.15.60:0/2308505391 >> 172.21.15.60:0/2308505391 conn(0x7fb0780fa330 msgr2=0x7fb0780fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:36.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:36.042+0000 7fb07e47f700 1 -- 172.21.15.60:0/2308505391 shutdown_connections 2023-07-20T01:27:36.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:36.042+0000 7fb07e47f700 1 -- 172.21.15.60:0/2308505391 wait complete. 2023-07-20T01:27:36.025 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 12 2023-07-20T01:27:36.443 INFO:journalctl@ceph.mds.b.smithi136.stdout:Jul 20 01:27:36 smithi136 systemd[1]: Starting Ceph mds.b for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:27:36.696 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:36 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/2308505391' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:36.696 INFO:journalctl@ceph.mds.b.smithi136.stdout:Jul 20 01:27:36 smithi136 podman[123154]: 2023-07-20T01:27:36.696 INFO:journalctl@ceph.mds.b.smithi136.stdout:Jul 20 01:27:36 smithi136 ceph-mds[123195]: -- 172.21.15.136:0/1171838230 <== mon.2 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56168dbf71e0 con 0x56168dbcd000 2023-07-20T01:27:36.924 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:36 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/2308505391' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:36.947 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:36 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/2308505391' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:37.144 INFO:journalctl@ceph.mds.b.smithi136.stdout:Jul 20 01:27:36 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-b[123171]: starting mds.b at 2023-07-20T01:27:37.144 INFO:journalctl@ceph.mds.b.smithi136.stdout:Jul 20 01:27:36 smithi136 bash[123154]: 898ceda19959031bdc44d44ac3864375f84ab4ecbe1fc5b3596d674247ee370e 2023-07-20T01:27:37.144 INFO:journalctl@ceph.mds.b.smithi136.stdout:Jul 20 01:27:36 smithi136 systemd[1]: Started Ceph mds.b for 7daf031e-269b-11ee-9b34-001a4aab830c. 2023-07-20T01:27:38.034 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-20T01:27:38.061 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:37 smithi060 ceph-mon[96524]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:38.061 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:37 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:38.062 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:37 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:38.062 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:37 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:38.062 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:37 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:38.062 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:37 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:38.062 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:37 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:27:38.062 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:37 smithi060 ceph-mon[96524]: mds.? [v2:172.21.15.136:6838/1778968612,v1:172.21.15.136:6839/1778968612] up:boot 2023-07-20T01:27:38.062 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:37 smithi060 ceph-mon[96524]: fsmap 12 up:standby 2023-07-20T01:27:38.063 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:37 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-20T01:27:38.144 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:37 smithi136 ceph-mon[98083]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:38.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:37 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:38.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:37 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:38.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:37 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:38.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:37 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:38.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:37 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:38.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:37 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:27:38.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:37 smithi136 ceph-mon[98083]: mds.? [v2:172.21.15.136:6838/1778968612,v1:172.21.15.136:6839/1778968612] up:boot 2023-07-20T01:27:38.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:37 smithi136 ceph-mon[98083]: fsmap 12 up:standby 2023-07-20T01:27:38.147 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:37 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-20T01:27:38.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.205+0000 7f1d698c7700 1 -- 172.21.15.60:0/636080834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d64103b10 msgr2=0x7f1d64105f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:38.183 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.205+0000 7f1d698c7700 1 --2- 172.21.15.60:0/636080834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d64103b10 0x7f1d64105f00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1d58009990 tx=0x7f1d58009ca0 comp rx=0 tx=0).stop 2023-07-20T01:27:38.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.205+0000 7f1d698c7700 1 -- 172.21.15.60:0/636080834 shutdown_connections 2023-07-20T01:27:38.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.205+0000 7f1d698c7700 1 --2- 172.21.15.60:0/636080834 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1d64106440 0x7f1d641088b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:38.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.205+0000 7f1d698c7700 1 --2- 172.21.15.60:0/636080834 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d641011e0 0x7f1d641035d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:38.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.205+0000 7f1d698c7700 1 --2- 172.21.15.60:0/636080834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d64103b10 0x7f1d64105f00 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:38.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.205+0000 7f1d698c7700 1 -- 172.21.15.60:0/636080834 >> 172.21.15.60:0/636080834 conn(0x7f1d640fabd0 msgr2=0x7f1d640fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:38.184 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.206+0000 7f1d698c7700 1 -- 172.21.15.60:0/636080834 shutdown_connections 2023-07-20T01:27:38.185 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.206+0000 7f1d698c7700 1 -- 172.21.15.60:0/636080834 wait complete. 2023-07-20T01:27:38.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.206+0000 7f1d698c7700 1 Processor -- start 2023-07-20T01:27:38.186 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.206+0000 7f1d698c7700 1 -- start start 2023-07-20T01:27:38.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.207+0000 7f1d698c7700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d641011e0 0x7f1d64100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:38.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.207+0000 7f1d698c7700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d64103b10 0x7f1d640ff190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:38.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.207+0000 7f1d698c7700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1d64106440 0x7f1d640ff6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:38.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.207+0000 7f1d698c7700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d640ffe70 con 0x7f1d64103b10 2023-07-20T01:27:38.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.207+0000 7f1d698c7700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d640fffe0 con 0x7f1d641011e0 2023-07-20T01:27:38.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.207+0000 7f1d698c7700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d641a03b0 con 0x7f1d64106440 2023-07-20T01:27:38.187 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.207+0000 7f1d627fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d64103b10 0x7f1d640ff190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:38.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.207+0000 7f1d62ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d641011e0 0x7f1d64100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:38.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.207+0000 7f1d627fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d64103b10 0x7f1d640ff190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43034/0 (socket says 172.21.15.60:43034) 2023-07-20T01:27:38.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.207+0000 7f1d627fc700 1 -- 172.21.15.60:0/3685527652 learned_addr learned my addr 172.21.15.60:0/3685527652 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:27:38.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.207+0000 7f1d637fe700 1 --2- 172.21.15.60:0/3685527652 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1d64106440 0x7f1d640ff6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:38.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.207+0000 7f1d62ffd700 1 -- 172.21.15.60:0/3685527652 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1d64106440 msgr2=0x7f1d640ff6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:38.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.207+0000 7f1d62ffd700 1 --2- 172.21.15.60:0/3685527652 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1d64106440 0x7f1d640ff6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:38.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.207+0000 7f1d62ffd700 1 -- 172.21.15.60:0/3685527652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d64103b10 msgr2=0x7f1d640ff190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:38.188 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.207+0000 7f1d62ffd700 1 --2- 172.21.15.60:0/3685527652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d64103b10 0x7f1d640ff190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:38.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.207+0000 7f1d62ffd700 1 -- 172.21.15.60:0/3685527652 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d580095f0 con 0x7f1d641011e0 2023-07-20T01:27:38.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.207+0000 7f1d627fc700 1 --2- 172.21.15.60:0/3685527652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d64103b10 0x7f1d640ff190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T01:27:38.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.207+0000 7f1d62ffd700 1 --2- 172.21.15.60:0/3685527652 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d641011e0 0x7f1d64100b50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1d5400e9f0 tx=0x7f1d5400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:38.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.207+0000 7f1d688c5700 1 -- 172.21.15.60:0/3685527652 <== mon.2 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5400c9c0 con 0x7f1d641011e0 2023-07-20T01:27:38.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.208+0000 7f1d688c5700 1 -- 172.21.15.60:0/3685527652 <== mon.2 v2:172.21.15.136:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1d54010430 con 0x7f1d641011e0 2023-07-20T01:27:38.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.208+0000 7f1d698c7700 1 -- 172.21.15.60:0/3685527652 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d641a0690 con 0x7f1d641011e0 2023-07-20T01:27:38.189 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.208+0000 7f1d688c5700 1 -- 172.21.15.60:0/3685527652 <== mon.2 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d54018680 con 0x7f1d641011e0 2023-07-20T01:27:38.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.208+0000 7f1d698c7700 1 -- 172.21.15.60:0/3685527652 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d641a0c70 con 0x7f1d641011e0 2023-07-20T01:27:38.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.209+0000 7f1d698c7700 1 -- 172.21.15.60:0/3685527652 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d640fc750 con 0x7f1d641011e0 2023-07-20T01:27:38.190 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.210+0000 7f1d688c5700 1 -- 172.21.15.60:0/3685527652 <== mon.2 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f1d540105a0 con 0x7f1d641011e0 2023-07-20T01:27:38.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.211+0000 7f1d688c5700 1 --2- 172.21.15.60:0/3685527652 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f1d4c0a1220 0x7f1d4c0a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:38.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.211+0000 7f1d627fc700 1 --2- 172.21.15.60:0/3685527652 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f1d4c0a1220 0x7f1d4c0a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:38.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.211+0000 7f1d688c5700 1 -- 172.21.15.60:0/3685527652 <== mon.2 v2:172.21.15.136:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f1d54014070 con 0x7f1d641011e0 2023-07-20T01:27:38.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.211+0000 7f1d627fc700 1 --2- 172.21.15.60:0/3685527652 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f1d4c0a1220 0x7f1d4c0a36e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1d58006010 tx=0x7f1d58005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:38.191 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.213+0000 7f1d688c5700 1 -- 172.21.15.60:0/3685527652 <== mon.2 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1d54096160 con 0x7f1d641011e0 2023-07-20T01:27:38.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:37 smithi183 ceph-mon[98477]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:38.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:37 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:38.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:37 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:38.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:37 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:38.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:37 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:38.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:37 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:38.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:37 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:27:38.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:37 smithi183 ceph-mon[98477]: mds.? [v2:172.21.15.136:6838/1778968612,v1:172.21.15.136:6839/1778968612] up:boot 2023-07-20T01:27:38.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:37 smithi183 ceph-mon[98477]: fsmap 12 up:standby 2023-07-20T01:27:38.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:37 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-20T01:27:38.446 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.469+0000 7f1d698c7700 1 -- 172.21.15.60:0/3685527652 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d6402c970 con 0x7f1d641011e0 2023-07-20T01:27:38.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.470+0000 7f1d688c5700 1 -- 172.21.15.60:0/3685527652 <== mon.2 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9607 (secure 0 0 0) 0x7f1d54099780 con 0x7f1d641011e0 2023-07-20T01:27:38.447 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:27:38.448 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2893530475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2893530475},{"type":"v1","addr":"172.21.15.60:6835","nonce":2893530475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/302154574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":302154574},{"type":"v1","addr":"172.21.15.60:6837","nonce":302154574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6839/3584660034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":3584660034},{"type":"v1","addr":"172.21.15.60:6839","nonce":3584660034}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6841/2905977980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6840","nonce":2905977980},{"type":"v1","addr":"172.21.15.60:6841","nonce":2905977980}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6833/3759595107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3759595107},{"type":"v1","addr":"172.21.15.183:6833","nonce":3759595107}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1706932385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1706932385},{"type":"v1","addr":"172.21.15.136:6833","nonce":1706932385}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6835/3379647260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":3379647260},{"type":"v1","addr":"172.21.15.183:6835","nonce":3379647260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24476,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3349527335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3349527335},{"type":"v1","addr":"172.21.15.136:6835","nonce":3349527335}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/3540765285","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6836","nonce":3540765285},{"type":"v1","addr":"172.21.15.183:6837","nonce":3540765285}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6839/2229055327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6838","nonce":2229055327},{"type":"v1","addr":"172.21.15.183:6839","nonce":2229055327}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24500,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/895972199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":895972199},{"type":"v1","addr":"172.21.15.136:6837","nonce":895972199}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24536,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/1778968612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1778968612},{"type":"v1","addr":"172.21.15.136:6839","nonce":1778968612}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[]} 2023-07-20T01:27:38.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.472+0000 7f1d698c7700 1 -- 172.21.15.60:0/3685527652 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f1d4c0a1220 msgr2=0x7f1d4c0a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:38.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.472+0000 7f1d698c7700 1 --2- 172.21.15.60:0/3685527652 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f1d4c0a1220 0x7f1d4c0a36e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1d58006010 tx=0x7f1d58005960 comp rx=0 tx=0).stop 2023-07-20T01:27:38.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.472+0000 7f1d698c7700 1 -- 172.21.15.60:0/3685527652 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d641011e0 msgr2=0x7f1d64100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:38.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.472+0000 7f1d698c7700 1 --2- 172.21.15.60:0/3685527652 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d641011e0 0x7f1d64100b50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1d5400e9f0 tx=0x7f1d5400ed00 comp rx=0 tx=0).stop 2023-07-20T01:27:38.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.472+0000 7f1d698c7700 1 -- 172.21.15.60:0/3685527652 shutdown_connections 2023-07-20T01:27:38.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.472+0000 7f1d698c7700 1 --2- 172.21.15.60:0/3685527652 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f1d4c0a1220 0x7f1d4c0a36e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:38.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.472+0000 7f1d698c7700 1 --2- 172.21.15.60:0/3685527652 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1d64106440 0x7f1d640ff6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:38.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.472+0000 7f1d698c7700 1 --2- 172.21.15.60:0/3685527652 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d641011e0 0x7f1d64100b50 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:38.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.472+0000 7f1d698c7700 1 --2- 172.21.15.60:0/3685527652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d64103b10 0x7f1d640ff190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:38.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.472+0000 7f1d698c7700 1 -- 172.21.15.60:0/3685527652 >> 172.21.15.60:0/3685527652 conn(0x7f1d640fabd0 msgr2=0x7f1d64101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:38.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.472+0000 7f1d698c7700 1 -- 172.21.15.60:0/3685527652 shutdown_connections 2023-07-20T01:27:38.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.472+0000 7f1d698c7700 1 -- 172.21.15.60:0/3685527652 wait complete. 2023-07-20T01:27:38.452 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 13 2023-07-20T01:27:38.462 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T01:27:38.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.655+0000 7f8888245700 1 -- 172.21.15.60:0/1974027282 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8880100970 msgr2=0x7f8880107220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:38.633 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.655+0000 7f8888245700 1 --2- 172.21.15.60:0/1974027282 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8880100970 0x7f8880107220 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f88780099c0 tx=0x7f8878009cd0 comp rx=0 tx=0).stop 2023-07-20T01:27:38.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.655+0000 7f8888245700 1 -- 172.21.15.60:0/1974027282 shutdown_connections 2023-07-20T01:27:38.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.655+0000 7f8888245700 1 --2- 172.21.15.60:0/1974027282 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8880100970 0x7f8880107220 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:38.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.655+0000 7f8888245700 1 --2- 172.21.15.60:0/1974027282 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f88800fede0 0x7f88800ff200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:38.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.655+0000 7f8888245700 1 --2- 172.21.15.60:0/1974027282 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f88800fffd0 0x7f8880100430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:38.634 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.655+0000 7f8888245700 1 -- 172.21.15.60:0/1974027282 >> 172.21.15.60:0/1974027282 conn(0x7f88800fa330 msgr2=0x7f88800fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:38.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.655+0000 7f8888245700 1 -- 172.21.15.60:0/1974027282 shutdown_connections 2023-07-20T01:27:38.635 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.655+0000 7f8888245700 1 -- 172.21.15.60:0/1974027282 wait complete. 2023-07-20T01:27:38.636 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.656+0000 7f8888245700 1 Processor -- start 2023-07-20T01:27:38.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.656+0000 7f8888245700 1 -- start start 2023-07-20T01:27:38.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.656+0000 7f8888245700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f88800fede0 0x7f88801962a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:38.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.656+0000 7f8888245700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f88800fffd0 0x7f88801967e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:38.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.656+0000 7f8888245700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8880100970 0x7f888019d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:38.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.656+0000 7f8888245700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888019de20 con 0x7f8880100970 2023-07-20T01:27:38.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.656+0000 7f8888245700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888019df60 con 0x7f88800fede0 2023-07-20T01:27:38.637 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.656+0000 7f8888245700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888019e0f0 con 0x7f88800fffd0 2023-07-20T01:27:38.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.656+0000 7f88867e2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8880100970 0x7f888019d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:38.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.656+0000 7f88867e2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8880100970 0x7f888019d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43048/0 (socket says 172.21.15.60:43048) 2023-07-20T01:27:38.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.656+0000 7f88867e2700 1 -- 172.21.15.60:0/2417502168 learned_addr learned my addr 172.21.15.60:0/2417502168 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:27:38.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.656+0000 7f8885fe1700 1 --2- 172.21.15.60:0/2417502168 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f88800fede0 0x7f88801962a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:38.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.656+0000 7f88857e0700 1 --2- 172.21.15.60:0/2417502168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f88800fffd0 0x7f88801967e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:38.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.657+0000 7f88867e2700 1 -- 172.21.15.60:0/2417502168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f88800fffd0 msgr2=0x7f88801967e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:38.638 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.657+0000 7f88867e2700 1 --2- 172.21.15.60:0/2417502168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f88800fffd0 0x7f88801967e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:38.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.657+0000 7f88867e2700 1 -- 172.21.15.60:0/2417502168 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f88800fede0 msgr2=0x7f88801962a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:38.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.657+0000 7f88867e2700 1 --2- 172.21.15.60:0/2417502168 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f88800fede0 0x7f88801962a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:38.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.657+0000 7f88867e2700 1 -- 172.21.15.60:0/2417502168 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88780095f0 con 0x7f8880100970 2023-07-20T01:27:38.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.657+0000 7f88867e2700 1 --2- 172.21.15.60:0/2417502168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8880100970 0x7f888019d850 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8878004730 tx=0x7f8878009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:38.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.657+0000 7f8876ffd700 1 -- 172.21.15.60:0/2417502168 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887801d070 con 0x7f8880100970 2023-07-20T01:27:38.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.657+0000 7f8876ffd700 1 -- 172.21.15.60:0/2417502168 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8878004000 con 0x7f8880100970 2023-07-20T01:27:38.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.657+0000 7f8888245700 1 -- 172.21.15.60:0/2417502168 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f888019e370 con 0x7f8880100970 2023-07-20T01:27:38.639 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.657+0000 7f8876ffd700 1 -- 172.21.15.60:0/2417502168 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8878005030 con 0x7f8880100970 2023-07-20T01:27:38.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.657+0000 7f8888245700 1 -- 172.21.15.60:0/2417502168 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f888019e7f0 con 0x7f8880100970 2023-07-20T01:27:38.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.658+0000 7f8888245700 1 -- 172.21.15.60:0/2417502168 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f888010ad30 con 0x7f8880100970 2023-07-20T01:27:38.640 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.659+0000 7f8876ffd700 1 -- 172.21.15.60:0/2417502168 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f88780047d0 con 0x7f8880100970 2023-07-20T01:27:38.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.660+0000 7f8876ffd700 1 --2- 172.21.15.60:0/2417502168 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f886c0a1220 0x7f886c0a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:38.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.660+0000 7f8876ffd700 1 -- 172.21.15.60:0/2417502168 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f88780cd6a0 con 0x7f8880100970 2023-07-20T01:27:38.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.660+0000 7f8885fe1700 1 --2- 172.21.15.60:0/2417502168 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f886c0a1220 0x7f886c0a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:38.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.660+0000 7f8885fe1700 1 --2- 172.21.15.60:0/2417502168 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f886c0a1220 0x7f886c0a36e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f88800ffe30 tx=0x7f887c00c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:38.641 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.663+0000 7f8876ffd700 1 -- 172.21.15.60:0/2417502168 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8878097320 con 0x7f8880100970 2023-07-20T01:27:38.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.883+0000 7f8888245700 1 -- 172.21.15.60:0/2417502168 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f888019ee30 con 0x7f8880100970 2023-07-20T01:27:38.865 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.887+0000 7f8876ffd700 1 -- 172.21.15.60:0/2417502168 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+18396 (secure 0 0 0) 0x7f887809a940 con 0x7f8880100970 2023-07-20T01:27:38.865 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:27:38.865 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":78,"fsid":"7daf031e-269b-11ee-9b34-001a4aab830c","created":"2023-07-20T01:20:36.943348+0000","modified":"2023-07-20T01:27:01.834100+0000","last_up_change":"2023-07-20T01:27:00.528800+0000","last_in_change":"2023-07-20T01:26:43.393378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T01:23:48.328825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9e5d1e7-1bfe-445f-8907-a1bf68b56c46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":452810841},{"type":"v1","addr":"172.21.15.60:6803","nonce":452810841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":452810841},{"type":"v1","addr":"172.21.15.60:6805","nonce":452810841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":452810841},{"type":"v1","addr":"172.21.15.60:6809","nonce":452810841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":452810841},{"type":"v1","addr":"172.21.15.60:6807","nonce":452810841}]},"public_addr":"172.21.15.60:6803/452810841","cluster_addr":"172.21.15.60:6805/452810841","heartbeat_back_addr":"172.21.15.60:6809/452810841","heartbeat_front_addr":"172.21.15.60:6807/452810841","state":["exists","up"]},{"osd":1,"uuid":"74dc92bd-d5ac-49ed-9918-3a91c53e9486","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":2541801111},{"type":"v1","addr":"172.21.15.136:6801","nonce":2541801111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":2541801111},{"type":"v1","addr":"172.21.15.136:6803","nonce":2541801111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":2541801111},{"type":"v1","addr":"172.21.15.136:6807","nonce":2541801111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":2541801111},{"type":"v1","addr":"172.21.15.136:6805","nonce":2541801111}]},"public_addr":"172.21.15.136:6801/2541801111","cluster_addr":"172.21.15.136:6803/2541801111","heartbeat_back_addr":"172.21.15.136:6807/2541801111","heartbeat_front_addr":"172.21.15.136:6805/2541801111","state":["exists","up"]},{"osd":2,"uuid":"c33a6742-8287-4fb4-b707-7e0d9660fd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":4233496322},{"type":"v1","addr":"172.21.15.183:6801","nonce":4233496322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":4233496322},{"type":"v1","addr":"172.21.15.183:6803","nonce":4233496322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":4233496322},{"type":"v1","addr":"172.21.15.183:6807","nonce":4233496322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":4233496322},{"type":"v1","addr":"172.21.15.183:6805","nonce":4233496322}]},"public_addr":"172.21.15.183:6801/4233496322","cluster_addr":"172.21.15.183:6803/4233496322","heartbeat_back_addr":"172.21.15.183:6807/4233496322","heartbeat_front_addr":"172.21.15.183:6805/4233496322","state":["exists","up"]},{"osd":3,"uuid":"bb99ac8e-19f3-4e76-907f-d1739cbddac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":3285594248},{"type":"v1","addr":"172.21.15.60:6811","nonce":3285594248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3285594248},{"type":"v1","addr":"172.21.15.60:6813","nonce":3285594248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":3285594248},{"type":"v1","addr":"172.21.15.60:6817","nonce":3285594248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3285594248},{"type":"v1","addr":"172.21.15.60:6815","nonce":3285594248}]},"public_addr":"172.21.15.60:6811/3285594248","cluster_addr":"172.21.15.60:6813/3285594248","heartbeat_back_addr":"172.21.15.60:6817/3285594248","heartbeat_front_addr":"172.21.15.60:6815/3285594248","state":["exists","up"]},{"osd":4,"uuid":"5a4e21c8-0151-445f-a94d-852a80b1bce6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2380508636},{"type":"v1","addr":"172.21.15.136:6809","nonce":2380508636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2380508636},{"type":"v1","addr":"172.21.15.136:6811","nonce":2380508636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2380508636},{"type":"v1","addr":"172.21.15.136:6815","nonce":2380508636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2380508636},{"type":"v1","addr":"172.21.15.136:6813","nonce":2380508636}]},"public_addr":"172.21.15.136:6809/2380508636","cluster_addr":"172.21.15.136:6811/2380508636","heartbeat_back_addr":"172.21.15.136:6815/2380508636","heartbeat_front_addr":"172.21.15.136:6813/2380508636","state":["exists","up"]},{"osd":5,"uuid":"a9e51980-2c9e-40ea-ad3f-cb233018455f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2401792837},{"type":"v1","addr":"172.21.15.183:6809","nonce":2401792837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2401792837},{"type":"v1","addr":"172.21.15.183:6811","nonce":2401792837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2401792837},{"type":"v1","addr":"172.21.15.183:6815","nonce":2401792837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2401792837},{"type":"v1","addr":"172.21.15.183:6813","nonce":2401792837}]},"public_addr":"172.21.15.183:6809/2401792837","cluster_addr":"172.21.15.183:6811/2401792837","heartbeat_back_addr":"172.21.15.183:6815/2401792837","heartbeat_front_addr":"172.21.15.183:6813/2401792837","state":["exists","up"]},{"osd":6,"uuid":"58ff5768-d850-40f2-b90c-1098dfdc3256","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":3834224571},{"type":"v1","addr":"172.21.15.60:6819","nonce":3834224571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":3834224571},{"type":"v1","addr":"172.21.15.60:6821","nonce":3834224571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":3834224571},{"type":"v1","addr":"172.21.15.60:6825","nonce":3834224571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":3834224571},{"type":"v1","addr":"172.21.15.60:6823","nonce":3834224571}]},"public_addr":"172.21.15.60:6819/3834224571","cluster_addr":"172.21.15.60:6821/3834224571","heartbeat_back_addr":"172.21.15.60:6825/3834224571","heartbeat_front_addr":"172.21.15.60:6823/3834224571","state":["exists","up"]},{"osd":7,"uuid":"375cd893-5f26-42bb-98ed-30b05911a7af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":404531062},{"type":"v1","addr":"172.21.15.136:6817","nonce":404531062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":404531062},{"type":"v1","addr":"172.21.15.136:6819","nonce":404531062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":404531062},{"type":"v1","addr":"172.21.15.136:6823","nonce":404531062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":404531062},{"type":"v1","addr":"172.21.15.136:6821","nonce":404531062}]},"public_addr":"172.21.15.136:6817/404531062","cluster_addr":"172.21.15.136:6819/404531062","heartbeat_back_addr":"172.21.15.136:6823/404531062","heartbeat_front_addr":"172.21.15.136:6821/404531062","state":["exists","up"]},{"osd":8,"uuid":"68b0051c-0fd3-4ea1-b7df-ea0b8409b8cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":502824503},{"type":"v1","addr":"172.21.15.183:6817","nonce":502824503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":502824503},{"type":"v1","addr":"172.21.15.183:6819","nonce":502824503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":502824503},{"type":"v1","addr":"172.21.15.183:6823","nonce":502824503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":502824503},{"type":"v1","addr":"172.21.15.183:6821","nonce":502824503}]},"public_addr":"172.21.15.183:6817/502824503","cluster_addr":"172.21.15.183:6819/502824503","heartbeat_back_addr":"172.21.15.183:6823/502824503","heartbeat_front_addr":"172.21.15.183:6821/502824503","state":["exists","up"]},{"osd":9,"uuid":"cafd4cc3-258c-4f88-8d3d-0eb4dfa2e2c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":634188302},{"type":"v1","addr":"172.21.15.60:6827","nonce":634188302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":634188302},{"type":"v1","addr":"172.21.15.60:6829","nonce":634188302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":634188302},{"type":"v1","addr":"172.21.15.60:6833","nonce":634188302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":634188302},{"type":"v1","addr":"172.21.15.60:6831","nonce":634188302}]},"public_addr":"172.21.15.60:6827/634188302","cluster_addr":"172.21.15.60:6829/634188302","heartbeat_back_addr":"172.21.15.60:6833/634188302","heartbeat_front_addr":"172.21.15.60:6831/634188302","state":["exists","up"]},{"osd":10,"uuid":"0b092b5a-8512-47d5-928b-995f8dc17353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":247446439},{"type":"v1","addr":"172.21.15.136:6825","nonce":247446439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":247446439},{"type":"v1","addr":"172.21.15.136:6827","nonce":247446439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":247446439},{"type":"v1","addr":"172.21.15.136:6831","nonce":247446439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":247446439},{"type":"v1","addr":"172.21.15.136:6829","nonce":247446439}]},"public_addr":"172.21.15.136:6825/247446439","cluster_addr":"172.21.15.136:6827/247446439","heartbeat_back_addr":"172.21.15.136:6831/247446439","heartbeat_front_addr":"172.21.15.136:6829/247446439","state":["exists","up"]},{"osd":11,"uuid":"8415b1a9-eb18-4b27-ad4f-9a676757bef9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1157339878},{"type":"v1","addr":"172.21.15.183:6825","nonce":1157339878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":1157339878},{"type":"v1","addr":"172.21.15.183:6827","nonce":1157339878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":1157339878},{"type":"v1","addr":"172.21.15.183:6831","nonce":1157339878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1157339878},{"type":"v1","addr":"172.21.15.183:6829","nonce":1157339878}]},"public_addr":"172.21.15.183:6825/1157339878","cluster_addr":"172.21.15.183:6827/1157339878","heartbeat_back_addr":"172.21.15.183:6831/1157339878","heartbeat_front_addr":"172.21.15.183:6829/1157339878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:23:03.185905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:23:24.597431+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:23:45.809074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:24:06.953357+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:24:28.808398+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:24:50.196615+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:25:11.554360+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:25:32.969005+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:25:54.487750+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:26:15.703232+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:26:37.390344+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:26:58.959419+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2364926895":"2023-07-21T01:21:37.803443+0000","172.21.15.60:6800/712953724":"2023-07-21T01:21:37.803443+0000","172.21.15.60:0/3180624926":"2023-07-21T01:21:37.803443+0000","172.21.15.60:6801/712953724":"2023-07-21T01:21:37.803443+0000","172.21.15.60:0/1758646915":"2023-07-21T01:21:11.418252+0000","172.21.15.60:6801/1628989585":"2023-07-21T01:21:11.418252+0000","172.21.15.60:0/884024563":"2023-07-21T01:21:11.418252+0000","172.21.15.60:0/296900825":"2023-07-21T01:21:11.418252+0000","172.21.15.60:6800/1628989585":"2023-07-21T01:21:11.418252+0000","172.21.15.60:0/3556060174":"2023-07-21T01:21:37.803443+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T01:27:38.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.890+0000 7f8888245700 1 -- 172.21.15.60:0/2417502168 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f886c0a1220 msgr2=0x7f886c0a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:38.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.890+0000 7f8888245700 1 --2- 172.21.15.60:0/2417502168 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f886c0a1220 0x7f886c0a36e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f88800ffe30 tx=0x7f887c00c040 comp rx=0 tx=0).stop 2023-07-20T01:27:38.867 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.890+0000 7f8888245700 1 -- 172.21.15.60:0/2417502168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8880100970 msgr2=0x7f888019d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:38.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.890+0000 7f8888245700 1 --2- 172.21.15.60:0/2417502168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8880100970 0x7f888019d850 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8878004730 tx=0x7f8878009580 comp rx=0 tx=0).stop 2023-07-20T01:27:38.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.890+0000 7f8888245700 1 -- 172.21.15.60:0/2417502168 shutdown_connections 2023-07-20T01:27:38.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.890+0000 7f8888245700 1 --2- 172.21.15.60:0/2417502168 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f886c0a1220 0x7f886c0a36e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:38.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.890+0000 7f8888245700 1 --2- 172.21.15.60:0/2417502168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8880100970 0x7f888019d850 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:38.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.890+0000 7f8888245700 1 --2- 172.21.15.60:0/2417502168 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f88800fede0 0x7f88801962a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:38.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.890+0000 7f8888245700 1 --2- 172.21.15.60:0/2417502168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f88800fffd0 0x7f88801967e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:38.868 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.890+0000 7f8888245700 1 -- 172.21.15.60:0/2417502168 >> 172.21.15.60:0/2417502168 conn(0x7f88800fa330 msgr2=0x7f8880108cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:38.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.890+0000 7f8888245700 1 -- 172.21.15.60:0/2417502168 shutdown_connections 2023-07-20T01:27:38.869 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:38.890+0000 7f8888245700 1 -- 172.21.15.60:0/2417502168 wait complete. 2023-07-20T01:27:38.879 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-20T01:23:48.328825+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-20T01:27:38.880 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-20T01:27:39.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.036+0000 7faa2beaf700 1 -- 172.21.15.60:0/2743711598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa24100880 msgr2=0x7faa24100ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:39.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.036+0000 7faa2beaf700 1 --2- 172.21.15.60:0/2743711598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa24100880 0x7faa24100ce0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7faa200099c0 tx=0x7faa20009cd0 comp rx=0 tx=0).stop 2023-07-20T01:27:39.014 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.036+0000 7faa2beaf700 1 -- 172.21.15.60:0/2743711598 shutdown_connections 2023-07-20T01:27:39.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.036+0000 7faa2beaf700 1 --2- 172.21.15.60:0/2743711598 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faa24101220 0x7faa24107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:39.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.036+0000 7faa2beaf700 1 --2- 172.21.15.60:0/2743711598 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa240ff690 0x7faa240ffab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:39.015 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.036+0000 7faa2beaf700 1 --2- 172.21.15.60:0/2743711598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa24100880 0x7faa24100ce0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:39.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.036+0000 7faa2beaf700 1 -- 172.21.15.60:0/2743711598 >> 172.21.15.60:0/2743711598 conn(0x7faa240fac00 msgr2=0x7faa240fd040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:39.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.036+0000 7faa2beaf700 1 -- 172.21.15.60:0/2743711598 shutdown_connections 2023-07-20T01:27:39.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.036+0000 7faa2beaf700 1 -- 172.21.15.60:0/2743711598 wait complete. 2023-07-20T01:27:39.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.037+0000 7faa2beaf700 1 Processor -- start 2023-07-20T01:27:39.016 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.037+0000 7faa2beaf700 1 -- start start 2023-07-20T01:27:39.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.037+0000 7faa2beaf700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa240ff690 0x7faa24069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:39.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.037+0000 7faa2beaf700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa24100880 0x7faa2419b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:39.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.037+0000 7faa2beaf700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faa24101220 0x7faa2419d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:39.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.037+0000 7faa2beaf700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2419dee0 con 0x7faa240ff690 2023-07-20T01:27:39.020 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.037+0000 7faa2beaf700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2419e050 con 0x7faa24100880 2023-07-20T01:27:39.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.037+0000 7faa2beaf700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2419e210 con 0x7faa24101220 2023-07-20T01:27:39.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.037+0000 7faa29c4b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa240ff690 0x7faa24069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:39.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.037+0000 7faa29c4b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa240ff690 0x7faa24069410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43058/0 (socket says 172.21.15.60:43058) 2023-07-20T01:27:39.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.037+0000 7faa29c4b700 1 -- 172.21.15.60:0/2039627613 learned_addr learned my addr 172.21.15.60:0/2039627613 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:27:39.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.037+0000 7faa2a44c700 1 --2- 172.21.15.60:0/2039627613 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faa24101220 0x7faa2419d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:39.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.037+0000 7faa2944a700 1 --2- 172.21.15.60:0/2039627613 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa24100880 0x7faa2419b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:39.021 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.037+0000 7faa29c4b700 1 -- 172.21.15.60:0/2039627613 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faa24101220 msgr2=0x7faa2419d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:39.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.037+0000 7faa29c4b700 1 --2- 172.21.15.60:0/2039627613 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faa24101220 0x7faa2419d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:39.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.037+0000 7faa29c4b700 1 -- 172.21.15.60:0/2039627613 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa24100880 msgr2=0x7faa2419b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:39.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.037+0000 7faa29c4b700 1 --2- 172.21.15.60:0/2039627613 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa24100880 0x7faa2419b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:39.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.037+0000 7faa29c4b700 1 -- 172.21.15.60:0/2039627613 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa200095f0 con 0x7faa240ff690 2023-07-20T01:27:39.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.037+0000 7faa2944a700 1 --2- 172.21.15.60:0/2039627613 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa24100880 0x7faa2419b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T01:27:39.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.038+0000 7faa29c4b700 1 --2- 172.21.15.60:0/2039627613 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa240ff690 0x7faa24069410 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7faa1400d980 tx=0x7faa1400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:39.022 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.038+0000 7faa1affd700 1 -- 172.21.15.60:0/2039627613 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1400b950 con 0x7faa240ff690 2023-07-20T01:27:39.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.038+0000 7faa1affd700 1 -- 172.21.15.60:0/2039627613 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7faa14007620 con 0x7faa240ff690 2023-07-20T01:27:39.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.038+0000 7faa2beaf700 1 -- 172.21.15.60:0/2039627613 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa2419e4f0 con 0x7faa240ff690 2023-07-20T01:27:39.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.038+0000 7faa1affd700 1 -- 172.21.15.60:0/2039627613 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1400f3c0 con 0x7faa240ff690 2023-07-20T01:27:39.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.038+0000 7faa2beaf700 1 -- 172.21.15.60:0/2039627613 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa2419ea00 con 0x7faa240ff690 2023-07-20T01:27:39.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.039+0000 7faa2beaf700 1 -- 172.21.15.60:0/2039627613 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa2410ad50 con 0x7faa240ff690 2023-07-20T01:27:39.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.040+0000 7faa1affd700 1 -- 172.21.15.60:0/2039627613 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7faa1400bab0 con 0x7faa240ff690 2023-07-20T01:27:39.023 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.040+0000 7faa1affd700 1 --2- 172.21.15.60:0/2039627613 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7faa100a1150 0x7faa100a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:39.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.040+0000 7faa1affd700 1 -- 172.21.15.60:0/2039627613 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7faa140ccba0 con 0x7faa240ff690 2023-07-20T01:27:39.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.040+0000 7faa2944a700 1 --2- 172.21.15.60:0/2039627613 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7faa100a1150 0x7faa100a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:39.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.041+0000 7faa2944a700 1 --2- 172.21.15.60:0/2039627613 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7faa100a1150 0x7faa100a3610 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7faa200095c0 tx=0x7faa20009550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:39.024 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.044+0000 7faa1affd700 1 -- 172.21.15.60:0/2039627613 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7faa140cce60 con 0x7faa240ff690 2023-07-20T01:27:39.144 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:38 smithi136 ceph-mon[98083]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:39.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:38 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:39.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:38 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/3685527652' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:39.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:38 smithi060 ceph-mon[96524]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:39.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:38 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:39.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:38 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/3685527652' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:39.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:38 smithi183 ceph-mon[98477]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:39.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:38 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:39.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:38 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/3685527652' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-20T01:27:39.237 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.260+0000 7faa2beaf700 1 -- 172.21.15.60:0/2039627613 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faa24106660 con 0x7faa240ff690 2023-07-20T01:27:39.238 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.260+0000 7faa1affd700 1 -- 172.21.15.60:0/2039627613 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7faa14096790 con 0x7faa240ff690 2023-07-20T01:27:39.238 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2023-07-20T01:27:39.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.262+0000 7faa2beaf700 1 -- 172.21.15.60:0/2039627613 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7faa100a1150 msgr2=0x7faa100a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:39.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.262+0000 7faa2beaf700 1 --2- 172.21.15.60:0/2039627613 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7faa100a1150 0x7faa100a3610 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7faa200095c0 tx=0x7faa20009550 comp rx=0 tx=0).stop 2023-07-20T01:27:39.240 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.262+0000 7faa2beaf700 1 -- 172.21.15.60:0/2039627613 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa240ff690 msgr2=0x7faa24069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:39.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.262+0000 7faa2beaf700 1 --2- 172.21.15.60:0/2039627613 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa240ff690 0x7faa24069410 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7faa1400d980 tx=0x7faa1400dc90 comp rx=0 tx=0).stop 2023-07-20T01:27:39.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.263+0000 7faa2beaf700 1 -- 172.21.15.60:0/2039627613 shutdown_connections 2023-07-20T01:27:39.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.263+0000 7faa2beaf700 1 --2- 172.21.15.60:0/2039627613 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7faa100a1150 0x7faa100a3610 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:39.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.263+0000 7faa2beaf700 1 --2- 172.21.15.60:0/2039627613 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faa24101220 0x7faa2419d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:39.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.263+0000 7faa2beaf700 1 --2- 172.21.15.60:0/2039627613 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa240ff690 0x7faa24069410 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:39.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.263+0000 7faa2beaf700 1 --2- 172.21.15.60:0/2039627613 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa24100880 0x7faa2419b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:39.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.263+0000 7faa2beaf700 1 -- 172.21.15.60:0/2039627613 >> 172.21.15.60:0/2039627613 conn(0x7faa240fac00 msgr2=0x7faa240fcf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:39.241 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.263+0000 7faa2beaf700 1 -- 172.21.15.60:0/2039627613 shutdown_connections 2023-07-20T01:27:39.242 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.263+0000 7faa2beaf700 1 -- 172.21.15.60:0/2039627613 wait complete. 2023-07-20T01:27:39.252 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-07-20T01:27:39.392 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.415+0000 7f0a1c28d700 1 -- 172.21.15.60:0/6933266 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a14101d80 msgr2=0x7f0a141021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:39.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.415+0000 7f0a1c28d700 1 --2- 172.21.15.60:0/6933266 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a14101d80 0x7f0a141021a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0a10009990 tx=0x7f0a10009ca0 comp rx=0 tx=0).stop 2023-07-20T01:27:39.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.415+0000 7f0a1c28d700 1 -- 172.21.15.60:0/6933266 shutdown_connections 2023-07-20T01:27:39.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.415+0000 7f0a1c28d700 1 --2- 172.21.15.60:0/6933266 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a140ff190 0x7f0a14108d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:39.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.415+0000 7f0a1c28d700 1 --2- 172.21.15.60:0/6933266 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a14101d80 0x7f0a141021a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:39.393 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.415+0000 7f0a1c28d700 1 --2- 172.21.15.60:0/6933266 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a140fe7f0 0x7f0a140fec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:39.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.415+0000 7f0a1c28d700 1 -- 172.21.15.60:0/6933266 >> 172.21.15.60:0/6933266 conn(0x7f0a140fa330 msgr2=0x7f0a140fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:39.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.415+0000 7f0a1c28d700 1 -- 172.21.15.60:0/6933266 shutdown_connections 2023-07-20T01:27:39.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.415+0000 7f0a1c28d700 1 -- 172.21.15.60:0/6933266 wait complete. 2023-07-20T01:27:39.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.415+0000 7f0a1c28d700 1 Processor -- start 2023-07-20T01:27:39.394 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.415+0000 7f0a1c28d700 1 -- start start 2023-07-20T01:27:39.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.416+0000 7f0a1c28d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a140fe7f0 0x7f0a141963e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:39.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.416+0000 7f0a1c28d700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a140ff190 0x7f0a14196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:39.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.416+0000 7f0a1c28d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a14101d80 0x7f0a1419d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:39.396 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.416+0000 7f0a1c28d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1419df60 con 0x7f0a140fe7f0 2023-07-20T01:27:39.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.416+0000 7f0a1c28d700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1419e0a0 con 0x7f0a14101d80 2023-07-20T01:27:39.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.416+0000 7f0a1c28d700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1419e230 con 0x7f0a140ff190 2023-07-20T01:27:39.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.416+0000 7f0a1a029700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a140fe7f0 0x7f0a141963e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:39.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.416+0000 7f0a1a029700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a140fe7f0 0x7f0a141963e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43072/0 (socket says 172.21.15.60:43072) 2023-07-20T01:27:39.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.416+0000 7f0a1a029700 1 -- 172.21.15.60:0/239056507 learned_addr learned my addr 172.21.15.60:0/239056507 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:27:39.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.416+0000 7f0a19828700 1 --2- 172.21.15.60:0/239056507 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a140ff190 0x7f0a14196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:39.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.416+0000 7f0a1a029700 1 -- 172.21.15.60:0/239056507 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a140ff190 msgr2=0x7f0a14196920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:39.397 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.416+0000 7f0a1a82a700 1 --2- 172.21.15.60:0/239056507 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a14101d80 0x7f0a1419d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:39.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.416+0000 7f0a1a029700 1 --2- 172.21.15.60:0/239056507 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a140ff190 0x7f0a14196920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:39.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.416+0000 7f0a1a029700 1 -- 172.21.15.60:0/239056507 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a14101d80 msgr2=0x7f0a1419d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:39.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.416+0000 7f0a1a029700 1 --2- 172.21.15.60:0/239056507 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a14101d80 0x7f0a1419d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:39.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.416+0000 7f0a1a029700 1 -- 172.21.15.60:0/239056507 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a100095f0 con 0x7f0a140fe7f0 2023-07-20T01:27:39.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.416+0000 7f0a1a82a700 1 --2- 172.21.15.60:0/239056507 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a14101d80 0x7f0a1419d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T01:27:39.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.416+0000 7f0a1a029700 1 --2- 172.21.15.60:0/239056507 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a140fe7f0 0x7f0a141963e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0a100059d0 tx=0x7f0a10005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:39.398 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.416+0000 7f0a0b7fe700 1 -- 172.21.15.60:0/239056507 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a1001d070 con 0x7f0a140fe7f0 2023-07-20T01:27:39.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.416+0000 7f0a0b7fe700 1 -- 172.21.15.60:0/239056507 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0a1000beb0 con 0x7f0a140fe7f0 2023-07-20T01:27:39.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.416+0000 7f0a1c28d700 1 -- 172.21.15.60:0/239056507 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a1419e4b0 con 0x7f0a140fe7f0 2023-07-20T01:27:39.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.416+0000 7f0a0b7fe700 1 -- 172.21.15.60:0/239056507 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a10022e90 con 0x7f0a140fe7f0 2023-07-20T01:27:39.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.416+0000 7f0a1c28d700 1 -- 172.21.15.60:0/239056507 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a1419e8d0 con 0x7f0a140fe7f0 2023-07-20T01:27:39.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.417+0000 7f0a1c28d700 1 -- 172.21.15.60:0/239056507 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a14108010 con 0x7f0a140fe7f0 2023-07-20T01:27:39.399 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.418+0000 7f0a0b7fe700 1 -- 172.21.15.60:0/239056507 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0a100225e0 con 0x7f0a140fe7f0 2023-07-20T01:27:39.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.419+0000 7f0a0b7fe700 1 --2- 172.21.15.60:0/239056507 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f0a000a11a0 0x7f0a000a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:39.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.419+0000 7f0a19828700 1 --2- 172.21.15.60:0/239056507 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f0a000a11a0 0x7f0a000a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:39.400 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.419+0000 7f0a0b7fe700 1 -- 172.21.15.60:0/239056507 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0a100cdc50 con 0x7f0a140fe7f0 2023-07-20T01:27:39.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.420+0000 7f0a19828700 1 --2- 172.21.15.60:0/239056507 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f0a000a11a0 0x7f0a000a3660 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0a04009bf0 tx=0x7f0a04009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:39.401 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.422+0000 7f0a0b7fe700 1 -- 172.21.15.60:0/239056507 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0a100978d0 con 0x7f0a140fe7f0 2023-07-20T01:27:39.609 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.631+0000 7f0a1c28d700 1 -- 172.21.15.60:0/239056507 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f0a1402c970 con 0x7f0a140fe7f0 2023-07-20T01:27:39.610 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.632+0000 7f0a0b7fe700 1 -- 172.21.15.60:0/239056507 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v78) v1 ==== 84+0+65 (secure 0 0 0) 0x7f0a1009aef0 con 0x7f0a140fe7f0 2023-07-20T01:27:39.610 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:27:39.610 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2023-07-20T01:27:39.611 INFO:teuthology.orchestra.run.smithi060.stdout: { 2023-07-20T01:27:39.611 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2023-07-20T01:27:39.611 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": ".mgr" 2023-07-20T01:27:39.611 INFO:teuthology.orchestra.run.smithi060.stdout: } 2023-07-20T01:27:39.612 INFO:teuthology.orchestra.run.smithi060.stdout:] 2023-07-20T01:27:39.614 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.634+0000 7f0a1c28d700 1 -- 172.21.15.60:0/239056507 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f0a000a11a0 msgr2=0x7f0a000a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:39.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.634+0000 7f0a1c28d700 1 --2- 172.21.15.60:0/239056507 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f0a000a11a0 0x7f0a000a3660 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0a04009bf0 tx=0x7f0a04009260 comp rx=0 tx=0).stop 2023-07-20T01:27:39.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.634+0000 7f0a1c28d700 1 -- 172.21.15.60:0/239056507 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a140fe7f0 msgr2=0x7f0a141963e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:39.615 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.634+0000 7f0a1c28d700 1 --2- 172.21.15.60:0/239056507 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a140fe7f0 0x7f0a141963e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0a100059d0 tx=0x7f0a10005e80 comp rx=0 tx=0).stop 2023-07-20T01:27:39.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.634+0000 7f0a1c28d700 1 -- 172.21.15.60:0/239056507 shutdown_connections 2023-07-20T01:27:39.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.634+0000 7f0a1c28d700 1 --2- 172.21.15.60:0/239056507 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f0a000a11a0 0x7f0a000a3660 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:39.616 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.634+0000 7f0a1c28d700 1 --2- 172.21.15.60:0/239056507 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a14101d80 0x7f0a1419d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:39.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.634+0000 7f0a1c28d700 1 --2- 172.21.15.60:0/239056507 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a140fe7f0 0x7f0a141963e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:39.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.634+0000 7f0a1c28d700 1 --2- 172.21.15.60:0/239056507 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a140ff190 0x7f0a14196920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:39.617 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.634+0000 7f0a1c28d700 1 -- 172.21.15.60:0/239056507 >> 172.21.15.60:0/239056507 conn(0x7f0a140fa330 msgr2=0x7f0a14105150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:39.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.634+0000 7f0a1c28d700 1 -- 172.21.15.60:0/239056507 shutdown_connections 2023-07-20T01:27:39.618 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.634+0000 7f0a1c28d700 1 -- 172.21.15.60:0/239056507 wait complete. 2023-07-20T01:27:39.626 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-07-20T01:27:39.626 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-07-20T01:27:39.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.783+0000 7f3057807700 1 -- 172.21.15.60:0/3461783946 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3050106440 msgr2=0x7f30501088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:39.761 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.783+0000 7f3057807700 1 --2- 172.21.15.60:0/3461783946 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3050106440 0x7f30501088b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3048009f90 tx=0x7f304800b330 comp rx=0 tx=0).stop 2023-07-20T01:27:39.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.784+0000 7f3057807700 1 -- 172.21.15.60:0/3461783946 shutdown_connections 2023-07-20T01:27:39.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.784+0000 7f3057807700 1 --2- 172.21.15.60:0/3461783946 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3050106440 0x7f30501088b0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:39.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.784+0000 7f3057807700 1 --2- 172.21.15.60:0/3461783946 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f30501011e0 0x7f30501035d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:39.762 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.784+0000 7f3057807700 1 --2- 172.21.15.60:0/3461783946 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3050103b10 0x7f3050105f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:39.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.784+0000 7f3057807700 1 -- 172.21.15.60:0/3461783946 >> 172.21.15.60:0/3461783946 conn(0x7f30500fabd0 msgr2=0x7f30500fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:39.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.784+0000 7f3057807700 1 -- 172.21.15.60:0/3461783946 shutdown_connections 2023-07-20T01:27:39.763 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.784+0000 7f3057807700 1 -- 172.21.15.60:0/3461783946 wait complete. 2023-07-20T01:27:39.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.784+0000 7f3057807700 1 Processor -- start 2023-07-20T01:27:39.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.785+0000 7f3057807700 1 -- start start 2023-07-20T01:27:39.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.785+0000 7f3057807700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f30501011e0 0x7f30501964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:39.765 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.785+0000 7f3057807700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3050103b10 0x7f3050196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:39.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.785+0000 7f3057807700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3050106440 0x7f305019da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:39.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.785+0000 7f3057807700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305019e040 con 0x7f3050106440 2023-07-20T01:27:39.766 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.785+0000 7f3057807700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305019e180 con 0x7f30501011e0 2023-07-20T01:27:39.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.785+0000 7f3057807700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305019e310 con 0x7f3050103b10 2023-07-20T01:27:39.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.785+0000 7f3055da4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3050106440 0x7f305019da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:39.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.785+0000 7f3055da4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3050106440 0x7f305019da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43090/0 (socket says 172.21.15.60:43090) 2023-07-20T01:27:39.767 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.785+0000 7f3055da4700 1 -- 172.21.15.60:0/3479154451 learned_addr learned my addr 172.21.15.60:0/3479154451 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:27:39.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.785+0000 7f30555a3700 1 --2- 172.21.15.60:0/3479154451 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f30501011e0 0x7f30501964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:39.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.785+0000 7f3055da4700 1 -- 172.21.15.60:0/3479154451 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3050103b10 msgr2=0x7f3050196a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:39.768 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.785+0000 7f3055da4700 1 --2- 172.21.15.60:0/3479154451 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3050103b10 0x7f3050196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:39.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.785+0000 7f3055da4700 1 -- 172.21.15.60:0/3479154451 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f30501011e0 msgr2=0x7f30501964c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:39.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.785+0000 7f3055da4700 1 --2- 172.21.15.60:0/3479154451 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f30501011e0 0x7f30501964c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:39.769 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.785+0000 7f3055da4700 1 -- 172.21.15.60:0/3479154451 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f304c0095f0 con 0x7f3050106440 2023-07-20T01:27:39.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.785+0000 7f3055da4700 1 --2- 172.21.15.60:0/3479154451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3050106440 0x7f305019da70 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3048007890 tx=0x7f3048008d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:39.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.785+0000 7f30467fc700 1 -- 172.21.15.60:0/3479154451 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304800e040 con 0x7f3050106440 2023-07-20T01:27:39.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.786+0000 7f3057807700 1 -- 172.21.15.60:0/3479154451 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3048009c40 con 0x7f3050106440 2023-07-20T01:27:39.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.786+0000 7f30467fc700 1 -- 172.21.15.60:0/3479154451 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f304800be00 con 0x7f3050106440 2023-07-20T01:27:39.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.786+0000 7f30467fc700 1 -- 172.21.15.60:0/3479154451 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30480052a0 con 0x7f3050106440 2023-07-20T01:27:39.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.786+0000 7f3057807700 1 -- 172.21.15.60:0/3479154451 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f305019e880 con 0x7f3050106440 2023-07-20T01:27:39.771 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.786+0000 7f3057807700 1 -- 172.21.15.60:0/3479154451 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30500fc750 con 0x7f3050106440 2023-07-20T01:27:39.772 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.788+0000 7f30467fc700 1 -- 172.21.15.60:0/3479154451 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f304801d470 con 0x7f3050106440 2023-07-20T01:27:39.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.788+0000 7f30467fc700 1 --2- 172.21.15.60:0/3479154451 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f303c0a1150 0x7f303c0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:39.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.788+0000 7f30555a3700 1 --2- 172.21.15.60:0/3479154451 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f303c0a1150 0x7f303c0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:39.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.788+0000 7f30467fc700 1 -- 172.21.15.60:0/3479154451 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f30480ce1c0 con 0x7f3050106440 2023-07-20T01:27:39.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.789+0000 7f30555a3700 1 --2- 172.21.15.60:0/3479154451 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f303c0a1150 0x7f303c0a3610 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f304c005fd0 tx=0x7f304c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:39.773 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.792+0000 7f30467fc700 1 -- 172.21.15.60:0/3479154451 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3048097e40 con 0x7f3050106440 2023-07-20T01:27:39.924 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:39 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/2417502168' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T01:27:39.924 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:39 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/2039627613' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T01:27:39.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:39 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/239056507' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-20T01:27:39.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:39 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:39.925 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:39 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:39.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:39 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:39.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:39 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:39.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:39 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:39.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:39 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:39.974 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:39.996+0000 7f3057807700 1 -- 172.21.15.60:0/3479154451 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f305002c970 con 0x7f3050106440 2023-07-20T01:27:40.144 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:39 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/2417502168' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T01:27:40.144 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:39 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/2039627613' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T01:27:40.144 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:39 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/239056507' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-20T01:27:40.144 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:39 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:40.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:39 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:40.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:39 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:40.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:39 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:40.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:39 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:40.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:39 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:40.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:39 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/2417502168' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T01:27:40.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:39 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/2039627613' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-20T01:27:40.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:39 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/239056507' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-20T01:27:40.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:39 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:40.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:39 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:40.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:39 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:40.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:39 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:40.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:39 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:40.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:39 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:40.873 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:40.896+0000 7f30467fc700 1 -- 172.21.15.60:0/3479154451 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v79) v1 ==== 138+0+0 (secure 0 0 0) 0x7f304809b460 con 0x7f3050106440 2023-07-20T01:27:40.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:40.900+0000 7f3057807700 1 -- 172.21.15.60:0/3479154451 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f303c0a1150 msgr2=0x7f303c0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:40.878 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:40.900+0000 7f3057807700 1 --2- 172.21.15.60:0/3479154451 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f303c0a1150 0x7f303c0a3610 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f304c005fd0 tx=0x7f304c009310 comp rx=0 tx=0).stop 2023-07-20T01:27:40.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:40.900+0000 7f3057807700 1 -- 172.21.15.60:0/3479154451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3050106440 msgr2=0x7f305019da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:40.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:40.900+0000 7f3057807700 1 --2- 172.21.15.60:0/3479154451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3050106440 0x7f305019da70 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3048007890 tx=0x7f3048008d30 comp rx=0 tx=0).stop 2023-07-20T01:27:40.879 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:40.902+0000 7f3057807700 1 -- 172.21.15.60:0/3479154451 shutdown_connections 2023-07-20T01:27:40.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:40.902+0000 7f3057807700 1 --2- 172.21.15.60:0/3479154451 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f303c0a1150 0x7f303c0a3610 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:40.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:40.902+0000 7f3057807700 1 --2- 172.21.15.60:0/3479154451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3050106440 0x7f305019da70 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:40.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:40.902+0000 7f3057807700 1 --2- 172.21.15.60:0/3479154451 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f30501011e0 0x7f30501964c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:40.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:40.902+0000 7f3057807700 1 --2- 172.21.15.60:0/3479154451 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3050103b10 0x7f3050196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:40.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:40.902+0000 7f3057807700 1 -- 172.21.15.60:0/3479154451 >> 172.21.15.60:0/3479154451 conn(0x7f30500fabd0 msgr2=0x7f3050101eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:40.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:40.902+0000 7f3057807700 1 -- 172.21.15.60:0/3479154451 shutdown_connections 2023-07-20T01:27:40.880 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:40.902+0000 7f3057807700 1 -- 172.21.15.60:0/3479154451 wait complete. 2023-07-20T01:27:40.881 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2023-07-20T01:27:40.892 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-07-20T01:27:41.033 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.052+0000 7f8c955c4700 1 -- 172.21.15.60:0/613675228 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c901007a0 msgr2=0x7f8c90100bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:41.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.052+0000 7f8c955c4700 1 --2- 172.21.15.60:0/613675228 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c901007a0 0x7f8c90100bc0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8c8000b0d0 tx=0x7f8c8000b3e0 comp rx=0 tx=0).stop 2023-07-20T01:27:41.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.052+0000 7f8c955c4700 1 -- 172.21.15.60:0/613675228 shutdown_connections 2023-07-20T01:27:41.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.052+0000 7f8c955c4700 1 --2- 172.21.15.60:0/613675228 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c90102330 0x7f8c90108be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:41.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.052+0000 7f8c955c4700 1 --2- 172.21.15.60:0/613675228 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c901007a0 0x7f8c90100bc0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:41.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.052+0000 7f8c955c4700 1 --2- 172.21.15.60:0/613675228 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c90101990 0x7f8c90101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:41.034 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.052+0000 7f8c955c4700 1 -- 172.21.15.60:0/613675228 >> 172.21.15.60:0/613675228 conn(0x7f8c900fbcf0 msgr2=0x7f8c900fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:41.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.052+0000 7f8c955c4700 1 -- 172.21.15.60:0/613675228 shutdown_connections 2023-07-20T01:27:41.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.052+0000 7f8c955c4700 1 -- 172.21.15.60:0/613675228 wait complete. 2023-07-20T01:27:41.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.053+0000 7f8c955c4700 1 Processor -- start 2023-07-20T01:27:41.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.053+0000 7f8c955c4700 1 -- start start 2023-07-20T01:27:41.035 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.053+0000 7f8c955c4700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c901007a0 0x7f8c90074520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:41.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.053+0000 7f8c955c4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c90101990 0x7f8c90072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:41.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.053+0000 7f8c955c4700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c90102330 0x7f8c900730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:41.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.053+0000 7f8c955c4700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c90073840 con 0x7f8c90101990 2023-07-20T01:27:41.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.053+0000 7f8c955c4700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c900739b0 con 0x7f8c90102330 2023-07-20T01:27:41.036 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.053+0000 7f8c955c4700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c901a0540 con 0x7f8c901007a0 2023-07-20T01:27:41.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.054+0000 7f8c8e7fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c90101990 0x7f8c90072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:41.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.054+0000 7f8c8f7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c90102330 0x7f8c900730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:41.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.054+0000 7f8c8f7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c90102330 0x7f8c900730a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.60:40958/0 (socket says 172.21.15.60:40958) 2023-07-20T01:27:41.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.054+0000 7f8c8effd700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c901007a0 0x7f8c90074520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:41.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.054+0000 7f8c8f7fe700 1 -- 172.21.15.60:0/186429974 learned_addr learned my addr 172.21.15.60:0/186429974 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:27:41.037 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.054+0000 7f8c8e7fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c90101990 0x7f8c90072b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43102/0 (socket says 172.21.15.60:43102) 2023-07-20T01:27:41.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.054+0000 7f8c8f7fe700 1 -- 172.21.15.60:0/186429974 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c901007a0 msgr2=0x7f8c90074520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:41.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.054+0000 7f8c8f7fe700 1 --2- 172.21.15.60:0/186429974 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c901007a0 0x7f8c90074520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:41.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.054+0000 7f8c8f7fe700 1 -- 172.21.15.60:0/186429974 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c90101990 msgr2=0x7f8c90072b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:41.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.054+0000 7f8c8f7fe700 1 --2- 172.21.15.60:0/186429974 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c90101990 0x7f8c90072b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:41.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.054+0000 7f8c8f7fe700 1 -- 172.21.15.60:0/186429974 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c80009d00 con 0x7f8c90102330 2023-07-20T01:27:41.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.054+0000 7f8c8e7fc700 1 --2- 172.21.15.60:0/186429974 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c90101990 0x7f8c90072b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T01:27:41.038 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.054+0000 7f8c8f7fe700 1 --2- 172.21.15.60:0/186429974 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c90102330 0x7f8c900730a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8c8800d980 tx=0x7f8c8800dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:41.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.054+0000 7f8c87fff700 1 -- 172.21.15.60:0/186429974 <== mon.2 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c88004a30 con 0x7f8c90102330 2023-07-20T01:27:41.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.054+0000 7f8c87fff700 1 -- 172.21.15.60:0/186429974 <== mon.2 v2:172.21.15.136:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8c880054b0 con 0x7f8c90102330 2023-07-20T01:27:41.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.054+0000 7f8c955c4700 1 -- 172.21.15.60:0/186429974 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c901a0820 con 0x7f8c90102330 2023-07-20T01:27:41.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.054+0000 7f8c87fff700 1 -- 172.21.15.60:0/186429974 <== mon.2 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8800fab0 con 0x7f8c90102330 2023-07-20T01:27:41.039 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.054+0000 7f8c955c4700 1 -- 172.21.15.60:0/186429974 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c901a0ce0 con 0x7f8c90102330 2023-07-20T01:27:41.040 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.055+0000 7f8c955c4700 1 -- 172.21.15.60:0/186429974 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c90107780 con 0x7f8c90102330 2023-07-20T01:27:41.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.056+0000 7f8c87fff700 1 -- 172.21.15.60:0/186429974 <== mon.2 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8c8800fc10 con 0x7f8c90102330 2023-07-20T01:27:41.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.057+0000 7f8c87fff700 1 --2- 172.21.15.60:0/186429974 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f8c7c0a1150 0x7f8c7c0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:41.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.057+0000 7f8c8effd700 1 --2- 172.21.15.60:0/186429974 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f8c7c0a1150 0x7f8c7c0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:41.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.057+0000 7f8c87fff700 1 -- 172.21.15.60:0/186429974 <== mon.2 v2:172.21.15.136:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f8c880cc820 con 0x7f8c90102330 2023-07-20T01:27:41.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.057+0000 7f8c8effd700 1 --2- 172.21.15.60:0/186429974 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f8c7c0a1150 0x7f8c7c0a3610 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8c800062a0 tx=0x7f8c8000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:41.041 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.061+0000 7f8c87fff700 1 -- 172.21.15.60:0/186429974 <== mon.2 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8c880962a0 con 0x7f8c90102330 2023-07-20T01:27:41.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:40 smithi136 ceph-mon[98083]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:41.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:40 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/3479154451' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-20T01:27:41.145 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:40 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:41.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:40 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:27:41.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:40 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:41.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:40 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:27:41.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:40 smithi060 ceph-mon[96524]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:41.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:40 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/3479154451' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-20T01:27:41.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:40 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:41.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:40 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:27:41.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:40 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:41.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:40 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:27:41.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:40 smithi183 ceph-mon[98477]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:41.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:40 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/3479154451' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-20T01:27:41.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:40 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:41.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:40 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:27:41.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:40 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:41.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:40 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-20T01:27:41.247 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.269+0000 7f8c955c4700 1 -- 172.21.15.60:0/186429974 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f8c900613b0 con 0x7f8c90102330 2023-07-20T01:27:41.886 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.902+0000 7f8c87fff700 1 -- 172.21.15.60:0/186429974 <== mon.2 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v80) v1 ==== 170+0+0 (secure 0 0 0) 0x7f8c880998c0 con 0x7f8c90102330 2023-07-20T01:27:41.894 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.917+0000 7f8c955c4700 1 -- 172.21.15.60:0/186429974 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f8c7c0a1150 msgr2=0x7f8c7c0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:41.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.917+0000 7f8c955c4700 1 --2- 172.21.15.60:0/186429974 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f8c7c0a1150 0x7f8c7c0a3610 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8c800062a0 tx=0x7f8c8000b040 comp rx=0 tx=0).stop 2023-07-20T01:27:41.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.917+0000 7f8c955c4700 1 -- 172.21.15.60:0/186429974 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c90102330 msgr2=0x7f8c900730a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:41.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.917+0000 7f8c955c4700 1 --2- 172.21.15.60:0/186429974 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c90102330 0x7f8c900730a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8c8800d980 tx=0x7f8c8800dc90 comp rx=0 tx=0).stop 2023-07-20T01:27:41.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.917+0000 7f8c955c4700 1 -- 172.21.15.60:0/186429974 shutdown_connections 2023-07-20T01:27:41.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.917+0000 7f8c955c4700 1 --2- 172.21.15.60:0/186429974 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f8c7c0a1150 0x7f8c7c0a3610 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:41.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.917+0000 7f8c955c4700 1 --2- 172.21.15.60:0/186429974 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c90102330 0x7f8c900730a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:41.895 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.917+0000 7f8c955c4700 1 --2- 172.21.15.60:0/186429974 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c901007a0 0x7f8c90074520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:41.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.917+0000 7f8c955c4700 1 --2- 172.21.15.60:0/186429974 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c90101990 0x7f8c90072b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:41.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.917+0000 7f8c955c4700 1 -- 172.21.15.60:0/186429974 >> 172.21.15.60:0/186429974 conn(0x7f8c900fbcf0 msgr2=0x7f8c900fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:41.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.918+0000 7f8c955c4700 1 -- 172.21.15.60:0/186429974 shutdown_connections 2023-07-20T01:27:41.896 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:41.918+0000 7f8c955c4700 1 -- 172.21.15.60:0/186429974 wait complete. 2023-07-20T01:27:41.897 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2023-07-20T01:27:41.909 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-07-20T01:27:42.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.073+0000 7f24f93b8700 1 -- 172.21.15.60:0/1838257046 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f24f4100850 msgr2=0x7f24f4100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:42.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.073+0000 7f24f93b8700 1 --2- 172.21.15.60:0/1838257046 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f24f4100850 0x7f24f4100cb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f24e80099c0 tx=0x7f24e8009cd0 comp rx=0 tx=0).stop 2023-07-20T01:27:42.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.074+0000 7f24f93b8700 1 -- 172.21.15.60:0/1838257046 shutdown_connections 2023-07-20T01:27:42.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.074+0000 7f24f93b8700 1 --2- 172.21.15.60:0/1838257046 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f24f41011f0 0x7f24f4107aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:42.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.074+0000 7f24f93b8700 1 --2- 172.21.15.60:0/1838257046 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f24f40ff660 0x7f24f40ffa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:42.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.074+0000 7f24f93b8700 1 --2- 172.21.15.60:0/1838257046 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f24f4100850 0x7f24f4100cb0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:42.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.074+0000 7f24f93b8700 1 -- 172.21.15.60:0/1838257046 >> 172.21.15.60:0/1838257046 conn(0x7f24f40fabd0 msgr2=0x7f24f40fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:42.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.074+0000 7f24f93b8700 1 -- 172.21.15.60:0/1838257046 shutdown_connections 2023-07-20T01:27:42.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.074+0000 7f24f93b8700 1 -- 172.21.15.60:0/1838257046 wait complete. 2023-07-20T01:27:42.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.074+0000 7f24f93b8700 1 Processor -- start 2023-07-20T01:27:42.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.074+0000 7f24f93b8700 1 -- start start 2023-07-20T01:27:42.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.075+0000 7f24f93b8700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f24f40ff660 0x7f24f41952f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:42.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.075+0000 7f24f93b8700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f24f4100850 0x7f24f4195830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:42.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.075+0000 7f24f93b8700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f24f41011f0 0x7f24f419c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:42.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.075+0000 7f24f93b8700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f419ce70 con 0x7f24f41011f0 2023-07-20T01:27:42.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.075+0000 7f24f93b8700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f419cfb0 con 0x7f24f40ff660 2023-07-20T01:27:42.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.075+0000 7f24f93b8700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f419d140 con 0x7f24f4100850 2023-07-20T01:27:42.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.075+0000 7f24f37fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f24f41011f0 0x7f24f419c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:42.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.075+0000 7f24f37fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f24f41011f0 0x7f24f419c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43120/0 (socket says 172.21.15.60:43120) 2023-07-20T01:27:42.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.075+0000 7f24f37fe700 1 -- 172.21.15.60:0/19132740 learned_addr learned my addr 172.21.15.60:0/19132740 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:27:42.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.075+0000 7f24f2ffd700 1 --2- 172.21.15.60:0/19132740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f24f40ff660 0x7f24f41952f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:42.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.075+0000 7f24f27fc700 1 --2- 172.21.15.60:0/19132740 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f24f4100850 0x7f24f4195830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:42.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.075+0000 7f24f37fe700 1 -- 172.21.15.60:0/19132740 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f24f4100850 msgr2=0x7f24f4195830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:42.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.075+0000 7f24f37fe700 1 --2- 172.21.15.60:0/19132740 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f24f4100850 0x7f24f4195830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:42.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.075+0000 7f24f37fe700 1 -- 172.21.15.60:0/19132740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f24f40ff660 msgr2=0x7f24f41952f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:42.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.075+0000 7f24f37fe700 1 --2- 172.21.15.60:0/19132740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f24f40ff660 0x7f24f41952f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:42.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.075+0000 7f24f37fe700 1 -- 172.21.15.60:0/19132740 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e80095f0 con 0x7f24f41011f0 2023-07-20T01:27:42.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.075+0000 7f24f37fe700 1 --2- 172.21.15.60:0/19132740 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f24f41011f0 0x7f24f419c8a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f24e400d980 tx=0x7f24e400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:42.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.075+0000 7f24dbfff700 1 -- 172.21.15.60:0/19132740 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e4004a30 con 0x7f24f41011f0 2023-07-20T01:27:42.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.075+0000 7f24dbfff700 1 -- 172.21.15.60:0/19132740 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f24e40054b0 con 0x7f24f41011f0 2023-07-20T01:27:42.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.075+0000 7f24f93b8700 1 -- 172.21.15.60:0/19132740 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24f419d420 con 0x7f24f41011f0 2023-07-20T01:27:42.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.075+0000 7f24dbfff700 1 -- 172.21.15.60:0/19132740 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e400fab0 con 0x7f24f41011f0 2023-07-20T01:27:42.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.075+0000 7f24f93b8700 1 -- 172.21.15.60:0/19132740 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24f419d8e0 con 0x7f24f41011f0 2023-07-20T01:27:42.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.076+0000 7f24f93b8700 1 -- 172.21.15.60:0/19132740 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24f4106d50 con 0x7f24f41011f0 2023-07-20T01:27:42.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.077+0000 7f24dbfff700 1 -- 172.21.15.60:0/19132740 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f24e400fc10 con 0x7f24f41011f0 2023-07-20T01:27:42.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.078+0000 7f24dbfff700 1 --2- 172.21.15.60:0/19132740 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f24dc0a1150 0x7f24dc0a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:42.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.078+0000 7f24f2ffd700 1 --2- 172.21.15.60:0/19132740 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f24dc0a1150 0x7f24dc0a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:42.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.078+0000 7f24dbfff700 1 -- 172.21.15.60:0/19132740 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f24e40cc920 con 0x7f24f41011f0 2023-07-20T01:27:42.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.079+0000 7f24f2ffd700 1 --2- 172.21.15.60:0/19132740 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f24dc0a1150 0x7f24dc0a3610 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f24ec009550 tx=0x7f24ec0091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:42.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.082+0000 7f24dbfff700 1 -- 172.21.15.60:0/19132740 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f24e4096220 con 0x7f24f41011f0 2023-07-20T01:27:42.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:41 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/3479154451' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-20T01:27:42.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:41 smithi060 ceph-mon[96524]: osdmap e79: 12 total, 12 up, 12 in 2023-07-20T01:27:42.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:41 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/186429974' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-20T01:27:42.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:41 smithi060 ceph-mon[96524]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-20T01:27:42.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:41 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/3479154451' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-20T01:27:42.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:41 smithi183 ceph-mon[98477]: osdmap e79: 12 total, 12 up, 12 in 2023-07-20T01:27:42.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:41 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/186429974' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-20T01:27:42.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:41 smithi183 ceph-mon[98477]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-20T01:27:42.323 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.345+0000 7f24f93b8700 1 -- 172.21.15.60:0/19132740 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f24f402cdf0 con 0x7f24f41011f0 2023-07-20T01:27:42.394 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:41 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/3479154451' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-20T01:27:42.394 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:41 smithi136 ceph-mon[98083]: osdmap e79: 12 total, 12 up, 12 in 2023-07-20T01:27:42.394 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:41 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/186429974' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-20T01:27:42.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:41 smithi136 ceph-mon[98083]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-20T01:27:42.900 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.922+0000 7f24dbfff700 1 -- 172.21.15.60:0/19132740 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f24f402cdf0 con 0x7f24f41011f0 2023-07-20T01:27:42.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.925+0000 7f24f93b8700 1 -- 172.21.15.60:0/19132740 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f24dc0a1150 msgr2=0x7f24dc0a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:42.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.925+0000 7f24f93b8700 1 --2- 172.21.15.60:0/19132740 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f24dc0a1150 0x7f24dc0a3610 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f24ec009550 tx=0x7f24ec0091b0 comp rx=0 tx=0).stop 2023-07-20T01:27:42.903 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.925+0000 7f24f93b8700 1 -- 172.21.15.60:0/19132740 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f24f41011f0 msgr2=0x7f24f419c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:42.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.925+0000 7f24f93b8700 1 --2- 172.21.15.60:0/19132740 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f24f41011f0 0x7f24f419c8a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f24e400d980 tx=0x7f24e400dc90 comp rx=0 tx=0).stop 2023-07-20T01:27:42.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.925+0000 7f24f93b8700 1 -- 172.21.15.60:0/19132740 shutdown_connections 2023-07-20T01:27:42.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.925+0000 7f24f93b8700 1 --2- 172.21.15.60:0/19132740 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f24dc0a1150 0x7f24dc0a3610 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:42.904 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.925+0000 7f24f93b8700 1 --2- 172.21.15.60:0/19132740 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f24f41011f0 0x7f24f419c8a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:42.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.925+0000 7f24f93b8700 1 --2- 172.21.15.60:0/19132740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f24f40ff660 0x7f24f41952f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:42.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.925+0000 7f24f93b8700 1 --2- 172.21.15.60:0/19132740 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f24f4100850 0x7f24f4195830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:42.905 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.925+0000 7f24f93b8700 1 -- 172.21.15.60:0/19132740 >> 172.21.15.60:0/19132740 conn(0x7f24f40fabd0 msgr2=0x7f24f4103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:42.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.925+0000 7f24f93b8700 1 -- 172.21.15.60:0/19132740 shutdown_connections 2023-07-20T01:27:42.906 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:42.925+0000 7f24f93b8700 1 -- 172.21.15.60:0/19132740 wait complete. 2023-07-20T01:27:42.909 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 2 and data pool 3 2023-07-20T01:27:42.921 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T01:27:43.052 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.075+0000 7f4158e07700 1 -- 172.21.15.60:0/833719056 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4154104300 msgr2=0x7f4154104780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:43.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.075+0000 7f4158e07700 1 --2- 172.21.15.60:0/833719056 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4154104300 0x7f4154104780 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f414c00b0a0 tx=0x7f414c00b3b0 comp rx=0 tx=0).stop 2023-07-20T01:27:43.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.075+0000 7f4158e07700 1 -- 172.21.15.60:0/833719056 shutdown_connections 2023-07-20T01:27:43.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.075+0000 7f4158e07700 1 --2- 172.21.15.60:0/833719056 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4154104300 0x7f4154104780 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:43.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.075+0000 7f4158e07700 1 --2- 172.21.15.60:0/833719056 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41540fefc0 0x7f41541013f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:43.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.075+0000 7f4158e07700 1 --2- 172.21.15.60:0/833719056 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4154101930 0x7f4154103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:43.053 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.075+0000 7f4158e07700 1 -- 172.21.15.60:0/833719056 >> 172.21.15.60:0/833719056 conn(0x7f41540faba0 msgr2=0x7f41540fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:43.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.075+0000 7f4158e07700 1 -- 172.21.15.60:0/833719056 shutdown_connections 2023-07-20T01:27:43.054 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.075+0000 7f4158e07700 1 -- 172.21.15.60:0/833719056 wait complete. 2023-07-20T01:27:43.055 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.076+0000 7f4158e07700 1 Processor -- start 2023-07-20T01:27:43.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.076+0000 7f4158e07700 1 -- start start 2023-07-20T01:27:43.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.076+0000 7f4158e07700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41540fefc0 0x7f415419ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:43.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.076+0000 7f4158e07700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4154101930 0x7f415419cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:43.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.076+0000 7f4158e07700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4154104300 0x7f41541a3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:43.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.076+0000 7f4158e07700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41541a4580 con 0x7f41540fefc0 2023-07-20T01:27:43.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.076+0000 7f4158e07700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41541a46c0 con 0x7f4154104300 2023-07-20T01:27:43.056 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.076+0000 7f4158e07700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41541a4850 con 0x7f4154101930 2023-07-20T01:27:43.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.076+0000 7f415259c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41540fefc0 0x7f415419ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:43.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.076+0000 7f415259c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41540fefc0 0x7f415419ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43134/0 (socket says 172.21.15.60:43134) 2023-07-20T01:27:43.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.076+0000 7f4151d9b700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4154101930 0x7f415419cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:43.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.076+0000 7f415259c700 1 -- 172.21.15.60:0/3569553728 learned_addr learned my addr 172.21.15.60:0/3569553728 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:27:43.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.076+0000 7f4152d9d700 1 --2- 172.21.15.60:0/3569553728 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4154104300 0x7f41541a3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:43.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.077+0000 7f415259c700 1 -- 172.21.15.60:0/3569553728 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4154101930 msgr2=0x7f415419cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:43.057 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.077+0000 7f415259c700 1 --2- 172.21.15.60:0/3569553728 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4154101930 0x7f415419cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:43.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.077+0000 7f415259c700 1 -- 172.21.15.60:0/3569553728 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4154104300 msgr2=0x7f41541a3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:43.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.077+0000 7f415259c700 1 --2- 172.21.15.60:0/3569553728 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4154104300 0x7f41541a3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:43.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.077+0000 7f415259c700 1 -- 172.21.15.60:0/3569553728 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f414c009d00 con 0x7f41540fefc0 2023-07-20T01:27:43.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.077+0000 7f4151d9b700 1 --2- 172.21.15.60:0/3569553728 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4154101930 0x7f415419cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-20T01:27:43.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.077+0000 7f415259c700 1 --2- 172.21.15.60:0/3569553728 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41540fefc0 0x7f415419ca00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f414400e9a0 tx=0x7f414400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:43.058 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.077+0000 7f414b7fe700 1 -- 172.21.15.60:0/3569553728 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4144009790 con 0x7f41540fefc0 2023-07-20T01:27:43.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.077+0000 7f414b7fe700 1 -- 172.21.15.60:0/3569553728 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4144010430 con 0x7f41540fefc0 2023-07-20T01:27:43.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.077+0000 7f4158e07700 1 -- 172.21.15.60:0/3569553728 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41541a4b30 con 0x7f41540fefc0 2023-07-20T01:27:43.059 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.077+0000 7f414b7fe700 1 -- 172.21.15.60:0/3569553728 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414400cea0 con 0x7f41540fefc0 2023-07-20T01:27:43.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.077+0000 7f4158e07700 1 -- 172.21.15.60:0/3569553728 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41541a5040 con 0x7f41540fefc0 2023-07-20T01:27:43.060 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.078+0000 7f4158e07700 1 -- 172.21.15.60:0/3569553728 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41540fc720 con 0x7f41540fefc0 2023-07-20T01:27:43.061 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.079+0000 7f414b7fe700 1 -- 172.21.15.60:0/3569553728 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f41440105a0 con 0x7f41540fefc0 2023-07-20T01:27:43.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.080+0000 7f414b7fe700 1 --2- 172.21.15.60:0/3569553728 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f41400a11a0 0x7f41400a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:43.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.080+0000 7f4151d9b700 1 --2- 172.21.15.60:0/3569553728 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f41400a11a0 0x7f41400a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:43.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.080+0000 7f414b7fe700 1 -- 172.21.15.60:0/3569553728 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f4144014070 con 0x7f41540fefc0 2023-07-20T01:27:43.062 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.080+0000 7f4151d9b700 1 --2- 172.21.15.60:0/3569553728 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f41400a11a0 0x7f41400a3660 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f413c011490 tx=0x7f413c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:43.063 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.083+0000 7f414b7fe700 1 -- 172.21.15.60:0/3569553728 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4144096430 con 0x7f41540fefc0 2023-07-20T01:27:43.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:42 smithi060 ceph-mon[96524]: pgmap v251: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:43.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:42 smithi060 ceph-mon[96524]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-20T01:27:43.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:42 smithi060 ceph-mon[96524]: osdmap e80: 12 total, 12 up, 12 in 2023-07-20T01:27:43.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:42 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/19132740' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-20T01:27:43.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:42 smithi060 ceph-mon[96524]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-20T01:27:43.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:42 smithi060 ceph-mon[96524]: osdmap e81: 12 total, 12 up, 12 in 2023-07-20T01:27:43.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:42 smithi183 ceph-mon[98477]: pgmap v251: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:43.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:42 smithi183 ceph-mon[98477]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-20T01:27:43.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:42 smithi183 ceph-mon[98477]: osdmap e80: 12 total, 12 up, 12 in 2023-07-20T01:27:43.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:42 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/19132740' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-20T01:27:43.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:42 smithi183 ceph-mon[98477]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-20T01:27:43.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:42 smithi183 ceph-mon[98477]: osdmap e81: 12 total, 12 up, 12 in 2023-07-20T01:27:43.294 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.317+0000 7f4158e07700 1 -- 172.21.15.60:0/3569553728 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f415402cc00 con 0x7f41540fefc0 2023-07-20T01:27:43.296 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.318+0000 7f414b7fe700 1 -- 172.21.15.60:0/3569553728 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+22106 (secure 0 0 0) 0x7f4144026090 con 0x7f41540fefc0 2023-07-20T01:27:43.296 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:27:43.296 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":81,"fsid":"7daf031e-269b-11ee-9b34-001a4aab830c","created":"2023-07-20T01:20:36.943348+0000","modified":"2023-07-20T01:27:42.347500+0000","last_up_change":"2023-07-20T01:27:00.528800+0000","last_in_change":"2023-07-20T01:26:43.393378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T01:23:48.328825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-20T01:27:40.434382+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-20T01:27:41.707917+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9e5d1e7-1bfe-445f-8907-a1bf68b56c46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":452810841},{"type":"v1","addr":"172.21.15.60:6803","nonce":452810841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":452810841},{"type":"v1","addr":"172.21.15.60:6805","nonce":452810841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":452810841},{"type":"v1","addr":"172.21.15.60:6809","nonce":452810841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":452810841},{"type":"v1","addr":"172.21.15.60:6807","nonce":452810841}]},"public_addr":"172.21.15.60:6803/452810841","cluster_addr":"172.21.15.60:6805/452810841","heartbeat_back_addr":"172.21.15.60:6809/452810841","heartbeat_front_addr":"172.21.15.60:6807/452810841","state":["exists","up"]},{"osd":1,"uuid":"74dc92bd-d5ac-49ed-9918-3a91c53e9486","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":2541801111},{"type":"v1","addr":"172.21.15.136:6801","nonce":2541801111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":2541801111},{"type":"v1","addr":"172.21.15.136:6803","nonce":2541801111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":2541801111},{"type":"v1","addr":"172.21.15.136:6807","nonce":2541801111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":2541801111},{"type":"v1","addr":"172.21.15.136:6805","nonce":2541801111}]},"public_addr":"172.21.15.136:6801/2541801111","cluster_addr":"172.21.15.136:6803/2541801111","heartbeat_back_addr":"172.21.15.136:6807/2541801111","heartbeat_front_addr":"172.21.15.136:6805/2541801111","state":["exists","up"]},{"osd":2,"uuid":"c33a6742-8287-4fb4-b707-7e0d9660fd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":4233496322},{"type":"v1","addr":"172.21.15.183:6801","nonce":4233496322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":4233496322},{"type":"v1","addr":"172.21.15.183:6803","nonce":4233496322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":4233496322},{"type":"v1","addr":"172.21.15.183:6807","nonce":4233496322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":4233496322},{"type":"v1","addr":"172.21.15.183:6805","nonce":4233496322}]},"public_addr":"172.21.15.183:6801/4233496322","cluster_addr":"172.21.15.183:6803/4233496322","heartbeat_back_addr":"172.21.15.183:6807/4233496322","heartbeat_front_addr":"172.21.15.183:6805/4233496322","state":["exists","up"]},{"osd":3,"uuid":"bb99ac8e-19f3-4e76-907f-d1739cbddac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":3285594248},{"type":"v1","addr":"172.21.15.60:6811","nonce":3285594248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3285594248},{"type":"v1","addr":"172.21.15.60:6813","nonce":3285594248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":3285594248},{"type":"v1","addr":"172.21.15.60:6817","nonce":3285594248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3285594248},{"type":"v1","addr":"172.21.15.60:6815","nonce":3285594248}]},"public_addr":"172.21.15.60:6811/3285594248","cluster_addr":"172.21.15.60:6813/3285594248","heartbeat_back_addr":"172.21.15.60:6817/3285594248","heartbeat_front_addr":"172.21.15.60:6815/3285594248","state":["exists","up"]},{"osd":4,"uuid":"5a4e21c8-0151-445f-a94d-852a80b1bce6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2380508636},{"type":"v1","addr":"172.21.15.136:6809","nonce":2380508636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2380508636},{"type":"v1","addr":"172.21.15.136:6811","nonce":2380508636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2380508636},{"type":"v1","addr":"172.21.15.136:6815","nonce":2380508636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2380508636},{"type":"v1","addr":"172.21.15.136:6813","nonce":2380508636}]},"public_addr":"172.21.15.136:6809/2380508636","cluster_addr":"172.21.15.136:6811/2380508636","heartbeat_back_addr":"172.21.15.136:6815/2380508636","heartbeat_front_addr":"172.21.15.136:6813/2380508636","state":["exists","up"]},{"osd":5,"uuid":"a9e51980-2c9e-40ea-ad3f-cb233018455f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2401792837},{"type":"v1","addr":"172.21.15.183:6809","nonce":2401792837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2401792837},{"type":"v1","addr":"172.21.15.183:6811","nonce":2401792837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2401792837},{"type":"v1","addr":"172.21.15.183:6815","nonce":2401792837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2401792837},{"type":"v1","addr":"172.21.15.183:6813","nonce":2401792837}]},"public_addr":"172.21.15.183:6809/2401792837","cluster_addr":"172.21.15.183:6811/2401792837","heartbeat_back_addr":"172.21.15.183:6815/2401792837","heartbeat_front_addr":"172.21.15.183:6813/2401792837","state":["exists","up"]},{"osd":6,"uuid":"58ff5768-d850-40f2-b90c-1098dfdc3256","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":3834224571},{"type":"v1","addr":"172.21.15.60:6819","nonce":3834224571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":3834224571},{"type":"v1","addr":"172.21.15.60:6821","nonce":3834224571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":3834224571},{"type":"v1","addr":"172.21.15.60:6825","nonce":3834224571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":3834224571},{"type":"v1","addr":"172.21.15.60:6823","nonce":3834224571}]},"public_addr":"172.21.15.60:6819/3834224571","cluster_addr":"172.21.15.60:6821/3834224571","heartbeat_back_addr":"172.21.15.60:6825/3834224571","heartbeat_front_addr":"172.21.15.60:6823/3834224571","state":["exists","up"]},{"osd":7,"uuid":"375cd893-5f26-42bb-98ed-30b05911a7af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":404531062},{"type":"v1","addr":"172.21.15.136:6817","nonce":404531062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":404531062},{"type":"v1","addr":"172.21.15.136:6819","nonce":404531062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":404531062},{"type":"v1","addr":"172.21.15.136:6823","nonce":404531062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":404531062},{"type":"v1","addr":"172.21.15.136:6821","nonce":404531062}]},"public_addr":"172.21.15.136:6817/404531062","cluster_addr":"172.21.15.136:6819/404531062","heartbeat_back_addr":"172.21.15.136:6823/404531062","heartbeat_front_addr":"172.21.15.136:6821/404531062","state":["exists","up"]},{"osd":8,"uuid":"68b0051c-0fd3-4ea1-b7df-ea0b8409b8cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":502824503},{"type":"v1","addr":"172.21.15.183:6817","nonce":502824503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":502824503},{"type":"v1","addr":"172.21.15.183:6819","nonce":502824503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":502824503},{"type":"v1","addr":"172.21.15.183:6823","nonce":502824503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":502824503},{"type":"v1","addr":"172.21.15.183:6821","nonce":502824503}]},"public_addr":"172.21.15.183:6817/502824503","cluster_addr":"172.21.15.183:6819/502824503","heartbeat_back_addr":"172.21.15.183:6823/502824503","heartbeat_front_addr":"172.21.15.183:6821/502824503","state":["exists","up"]},{"osd":9,"uuid":"cafd4cc3-258c-4f88-8d3d-0eb4dfa2e2c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":634188302},{"type":"v1","addr":"172.21.15.60:6827","nonce":634188302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":634188302},{"type":"v1","addr":"172.21.15.60:6829","nonce":634188302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":634188302},{"type":"v1","addr":"172.21.15.60:6833","nonce":634188302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":634188302},{"type":"v1","addr":"172.21.15.60:6831","nonce":634188302}]},"public_addr":"172.21.15.60:6827/634188302","cluster_addr":"172.21.15.60:6829/634188302","heartbeat_back_addr":"172.21.15.60:6833/634188302","heartbeat_front_addr":"172.21.15.60:6831/634188302","state":["exists","up"]},{"osd":10,"uuid":"0b092b5a-8512-47d5-928b-995f8dc17353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":247446439},{"type":"v1","addr":"172.21.15.136:6825","nonce":247446439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":247446439},{"type":"v1","addr":"172.21.15.136:6827","nonce":247446439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":247446439},{"type":"v1","addr":"172.21.15.136:6831","nonce":247446439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":247446439},{"type":"v1","addr":"172.21.15.136:6829","nonce":247446439}]},"public_addr":"172.21.15.136:6825/247446439","cluster_addr":"172.21.15.136:6827/247446439","heartbeat_back_addr":"172.21.15.136:6831/247446439","heartbeat_front_addr":"172.21.15.136:6829/247446439","state":["exists","up"]},{"osd":11,"uuid":"8415b1a9-eb18-4b27-ad4f-9a676757bef9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1157339878},{"type":"v1","addr":"172.21.15.183:6825","nonce":1157339878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":1157339878},{"type":"v1","addr":"172.21.15.183:6827","nonce":1157339878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":1157339878},{"type":"v1","addr":"172.21.15.183:6831","nonce":1157339878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1157339878},{"type":"v1","addr":"172.21.15.183:6829","nonce":1157339878}]},"public_addr":"172.21.15.183:6825/1157339878","cluster_addr":"172.21.15.183:6827/1157339878","heartbeat_back_addr":"172.21.15.183:6831/1157339878","heartbeat_front_addr":"172.21.15.183:6829/1157339878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:23:03.185905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:23:24.597431+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:23:45.809074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:24:06.953357+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:24:28.808398+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:24:50.196615+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:25:11.554360+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:25:32.969005+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:25:54.487750+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:26:15.703232+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:26:37.390344+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:26:58.959419+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2364926895":"2023-07-21T01:21:37.803443+0000","172.21.15.60:6800/712953724":"2023-07-21T01:21:37.803443+0000","172.21.15.60:0/3180624926":"2023-07-21T01:21:37.803443+0000","172.21.15.60:6801/712953724":"2023-07-21T01:21:37.803443+0000","172.21.15.60:0/1758646915":"2023-07-21T01:21:11.418252+0000","172.21.15.60:6801/1628989585":"2023-07-21T01:21:11.418252+0000","172.21.15.60:0/884024563":"2023-07-21T01:21:11.418252+0000","172.21.15.60:0/296900825":"2023-07-21T01:21:11.418252+0000","172.21.15.60:6800/1628989585":"2023-07-21T01:21:11.418252+0000","172.21.15.60:0/3556060174":"2023-07-21T01:21:37.803443+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T01:27:43.298 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.321+0000 7f4158e07700 1 -- 172.21.15.60:0/3569553728 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f41400a11a0 msgr2=0x7f41400a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:43.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.321+0000 7f4158e07700 1 --2- 172.21.15.60:0/3569553728 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f41400a11a0 0x7f41400a3660 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f413c011490 tx=0x7f413c009260 comp rx=0 tx=0).stop 2023-07-20T01:27:43.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.321+0000 7f4158e07700 1 -- 172.21.15.60:0/3569553728 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41540fefc0 msgr2=0x7f415419ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:43.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.321+0000 7f4158e07700 1 --2- 172.21.15.60:0/3569553728 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41540fefc0 0x7f415419ca00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f414400e9a0 tx=0x7f414400ecb0 comp rx=0 tx=0).stop 2023-07-20T01:27:43.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.321+0000 7f4158e07700 1 -- 172.21.15.60:0/3569553728 shutdown_connections 2023-07-20T01:27:43.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.321+0000 7f4158e07700 1 --2- 172.21.15.60:0/3569553728 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f41400a11a0 0x7f41400a3660 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:43.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.321+0000 7f4158e07700 1 --2- 172.21.15.60:0/3569553728 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4154104300 0x7f41541a3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:43.299 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.321+0000 7f4158e07700 1 --2- 172.21.15.60:0/3569553728 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41540fefc0 0x7f415419ca00 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:43.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.321+0000 7f4158e07700 1 --2- 172.21.15.60:0/3569553728 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4154101930 0x7f415419cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:43.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.321+0000 7f4158e07700 1 -- 172.21.15.60:0/3569553728 >> 172.21.15.60:0/3569553728 conn(0x7f41540faba0 msgr2=0x7f41540ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:43.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.321+0000 7f4158e07700 1 -- 172.21.15.60:0/3569553728 shutdown_connections 2023-07-20T01:27:43.300 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.321+0000 7f4158e07700 1 -- 172.21.15.60:0/3569553728 wait complete. 2023-07-20T01:27:43.313 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-20T01:27:43.394 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:42 smithi136 ceph-mon[98083]: pgmap v251: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-20T01:27:43.394 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:42 smithi136 ceph-mon[98083]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-20T01:27:43.394 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:42 smithi136 ceph-mon[98083]: osdmap e80: 12 total, 12 up, 12 in 2023-07-20T01:27:43.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:42 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/19132740' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-20T01:27:43.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:42 smithi136 ceph-mon[98083]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-20T01:27:43.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:42 smithi136 ceph-mon[98083]: osdmap e81: 12 total, 12 up, 12 in 2023-07-20T01:27:43.443 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.466+0000 7fcefeec9700 1 -- 172.21.15.60:0/983153424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcef8100790 msgr2=0x7fcef8100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:43.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.466+0000 7fcefeec9700 1 --2- 172.21.15.60:0/983153424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcef8100790 0x7fcef8100bb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fceec0099c0 tx=0x7fceec009cd0 comp rx=0 tx=0).stop 2023-07-20T01:27:43.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.466+0000 7fcefeec9700 1 -- 172.21.15.60:0/983153424 shutdown_connections 2023-07-20T01:27:43.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.466+0000 7fcefeec9700 1 --2- 172.21.15.60:0/983153424 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcef8102320 0x7fcef8108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:43.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.466+0000 7fcefeec9700 1 --2- 172.21.15.60:0/983153424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcef8100790 0x7fcef8100bb0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:43.444 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.466+0000 7fcefeec9700 1 --2- 172.21.15.60:0/983153424 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcef8101980 0x7fcef8101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:43.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.466+0000 7fcefeec9700 1 -- 172.21.15.60:0/983153424 >> 172.21.15.60:0/983153424 conn(0x7fcef80fbd20 msgr2=0x7fcef80fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:43.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.466+0000 7fcefeec9700 1 -- 172.21.15.60:0/983153424 shutdown_connections 2023-07-20T01:27:43.445 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.466+0000 7fcefeec9700 1 -- 172.21.15.60:0/983153424 wait complete. 2023-07-20T01:27:43.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.467+0000 7fcefeec9700 1 Processor -- start 2023-07-20T01:27:43.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.467+0000 7fcefeec9700 1 -- start start 2023-07-20T01:27:43.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.467+0000 7fcefeec9700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcef8100790 0x7fcef8196410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:43.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.467+0000 7fcefeec9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcef8101980 0x7fcef8196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:43.447 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.467+0000 7fcefeec9700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcef8102320 0x7fcef819d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:43.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.467+0000 7fcefeec9700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef819df90 con 0x7fcef8101980 2023-07-20T01:27:43.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.467+0000 7fcefeec9700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef819e0d0 con 0x7fcef8102320 2023-07-20T01:27:43.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.467+0000 7fcefeec9700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef819e260 con 0x7fcef8100790 2023-07-20T01:27:43.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.467+0000 7fcef7fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcef8101980 0x7fcef8196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:43.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.467+0000 7fcef7fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcef8101980 0x7fcef8196950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43150/0 (socket says 172.21.15.60:43150) 2023-07-20T01:27:43.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.467+0000 7fcef7fff700 1 -- 172.21.15.60:0/2427076588 learned_addr learned my addr 172.21.15.60:0/2427076588 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:27:43.448 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.467+0000 7fcefd466700 1 --2- 172.21.15.60:0/2427076588 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcef8102320 0x7fcef819d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:43.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.467+0000 7fcefcc65700 1 --2- 172.21.15.60:0/2427076588 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcef8100790 0x7fcef8196410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:43.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.467+0000 7fcef7fff700 1 -- 172.21.15.60:0/2427076588 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcef8100790 msgr2=0x7fcef8196410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:43.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.467+0000 7fcef7fff700 1 --2- 172.21.15.60:0/2427076588 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcef8100790 0x7fcef8196410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:43.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.467+0000 7fcef7fff700 1 -- 172.21.15.60:0/2427076588 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcef8102320 msgr2=0x7fcef819d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:43.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.467+0000 7fcef7fff700 1 --2- 172.21.15.60:0/2427076588 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcef8102320 0x7fcef819d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:43.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.467+0000 7fcef7fff700 1 -- 172.21.15.60:0/2427076588 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceec0095f0 con 0x7fcef8101980 2023-07-20T01:27:43.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.467+0000 7fcefcc65700 1 --2- 172.21.15.60:0/2427076588 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcef8100790 0x7fcef8196410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T01:27:43.449 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.468+0000 7fcef7fff700 1 --2- 172.21.15.60:0/2427076588 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcef8101980 0x7fcef8196950 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fcee400b530 tx=0x7fcee400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:43.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.468+0000 7fcef5ffb700 1 -- 172.21.15.60:0/2427076588 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee4011830 con 0x7fcef8101980 2023-07-20T01:27:43.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.468+0000 7fcef5ffb700 1 -- 172.21.15.60:0/2427076588 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fcee4011e20 con 0x7fcef8101980 2023-07-20T01:27:43.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.468+0000 7fcefeec9700 1 -- 172.21.15.60:0/2427076588 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcef819e540 con 0x7fcef8101980 2023-07-20T01:27:43.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.468+0000 7fcef5ffb700 1 -- 172.21.15.60:0/2427076588 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee400f2e0 con 0x7fcef8101980 2023-07-20T01:27:43.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.468+0000 7fcefeec9700 1 -- 172.21.15.60:0/2427076588 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcef819eaa0 con 0x7fcef8101980 2023-07-20T01:27:43.450 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.469+0000 7fcefeec9700 1 -- 172.21.15.60:0/2427076588 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcef810bf20 con 0x7fcef8101980 2023-07-20T01:27:43.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.470+0000 7fcef5ffb700 1 -- 172.21.15.60:0/2427076588 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fcee4011990 con 0x7fcef8101980 2023-07-20T01:27:43.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.470+0000 7fcef5ffb700 1 --2- 172.21.15.60:0/2427076588 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fcee80a1220 0x7fcee80a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:43.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.470+0000 7fcefcc65700 1 --2- 172.21.15.60:0/2427076588 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fcee80a1220 0x7fcee80a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:43.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.470+0000 7fcef5ffb700 1 -- 172.21.15.60:0/2427076588 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fcee40cc310 con 0x7fcef8101980 2023-07-20T01:27:43.451 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.471+0000 7fcefcc65700 1 --2- 172.21.15.60:0/2427076588 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fcee80a1220 0x7fcee80a36e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fceec006010 tx=0x7fceec000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:43.452 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.475+0000 7fcef5ffb700 1 -- 172.21.15.60:0/2427076588 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcee4095b10 con 0x7fcef8101980 2023-07-20T01:27:43.660 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.682+0000 7fcefeec9700 1 -- 172.21.15.60:0/2427076588 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcef810c110 con 0x7fcef8101980 2023-07-20T01:27:43.663 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.684+0000 7fcef5ffb700 1 -- 172.21.15.60:0/2427076588 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+22106 (secure 0 0 0) 0x7fcee4099130 con 0x7fcef8101980 2023-07-20T01:27:43.663 INFO:teuthology.orchestra.run.smithi060.stdout: 2023-07-20T01:27:43.664 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":81,"fsid":"7daf031e-269b-11ee-9b34-001a4aab830c","created":"2023-07-20T01:20:36.943348+0000","modified":"2023-07-20T01:27:42.347500+0000","last_up_change":"2023-07-20T01:27:00.528800+0000","last_in_change":"2023-07-20T01:26:43.393378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-20T01:23:48.328825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-20T01:27:40.434382+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-20T01:27:41.707917+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9e5d1e7-1bfe-445f-8907-a1bf68b56c46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":452810841},{"type":"v1","addr":"172.21.15.60:6803","nonce":452810841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":452810841},{"type":"v1","addr":"172.21.15.60:6805","nonce":452810841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":452810841},{"type":"v1","addr":"172.21.15.60:6809","nonce":452810841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":452810841},{"type":"v1","addr":"172.21.15.60:6807","nonce":452810841}]},"public_addr":"172.21.15.60:6803/452810841","cluster_addr":"172.21.15.60:6805/452810841","heartbeat_back_addr":"172.21.15.60:6809/452810841","heartbeat_front_addr":"172.21.15.60:6807/452810841","state":["exists","up"]},{"osd":1,"uuid":"74dc92bd-d5ac-49ed-9918-3a91c53e9486","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":2541801111},{"type":"v1","addr":"172.21.15.136:6801","nonce":2541801111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":2541801111},{"type":"v1","addr":"172.21.15.136:6803","nonce":2541801111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":2541801111},{"type":"v1","addr":"172.21.15.136:6807","nonce":2541801111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":2541801111},{"type":"v1","addr":"172.21.15.136:6805","nonce":2541801111}]},"public_addr":"172.21.15.136:6801/2541801111","cluster_addr":"172.21.15.136:6803/2541801111","heartbeat_back_addr":"172.21.15.136:6807/2541801111","heartbeat_front_addr":"172.21.15.136:6805/2541801111","state":["exists","up"]},{"osd":2,"uuid":"c33a6742-8287-4fb4-b707-7e0d9660fd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":4233496322},{"type":"v1","addr":"172.21.15.183:6801","nonce":4233496322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":4233496322},{"type":"v1","addr":"172.21.15.183:6803","nonce":4233496322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":4233496322},{"type":"v1","addr":"172.21.15.183:6807","nonce":4233496322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":4233496322},{"type":"v1","addr":"172.21.15.183:6805","nonce":4233496322}]},"public_addr":"172.21.15.183:6801/4233496322","cluster_addr":"172.21.15.183:6803/4233496322","heartbeat_back_addr":"172.21.15.183:6807/4233496322","heartbeat_front_addr":"172.21.15.183:6805/4233496322","state":["exists","up"]},{"osd":3,"uuid":"bb99ac8e-19f3-4e76-907f-d1739cbddac2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":3285594248},{"type":"v1","addr":"172.21.15.60:6811","nonce":3285594248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3285594248},{"type":"v1","addr":"172.21.15.60:6813","nonce":3285594248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":3285594248},{"type":"v1","addr":"172.21.15.60:6817","nonce":3285594248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3285594248},{"type":"v1","addr":"172.21.15.60:6815","nonce":3285594248}]},"public_addr":"172.21.15.60:6811/3285594248","cluster_addr":"172.21.15.60:6813/3285594248","heartbeat_back_addr":"172.21.15.60:6817/3285594248","heartbeat_front_addr":"172.21.15.60:6815/3285594248","state":["exists","up"]},{"osd":4,"uuid":"5a4e21c8-0151-445f-a94d-852a80b1bce6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2380508636},{"type":"v1","addr":"172.21.15.136:6809","nonce":2380508636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2380508636},{"type":"v1","addr":"172.21.15.136:6811","nonce":2380508636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2380508636},{"type":"v1","addr":"172.21.15.136:6815","nonce":2380508636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2380508636},{"type":"v1","addr":"172.21.15.136:6813","nonce":2380508636}]},"public_addr":"172.21.15.136:6809/2380508636","cluster_addr":"172.21.15.136:6811/2380508636","heartbeat_back_addr":"172.21.15.136:6815/2380508636","heartbeat_front_addr":"172.21.15.136:6813/2380508636","state":["exists","up"]},{"osd":5,"uuid":"a9e51980-2c9e-40ea-ad3f-cb233018455f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2401792837},{"type":"v1","addr":"172.21.15.183:6809","nonce":2401792837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2401792837},{"type":"v1","addr":"172.21.15.183:6811","nonce":2401792837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":2401792837},{"type":"v1","addr":"172.21.15.183:6815","nonce":2401792837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":2401792837},{"type":"v1","addr":"172.21.15.183:6813","nonce":2401792837}]},"public_addr":"172.21.15.183:6809/2401792837","cluster_addr":"172.21.15.183:6811/2401792837","heartbeat_back_addr":"172.21.15.183:6815/2401792837","heartbeat_front_addr":"172.21.15.183:6813/2401792837","state":["exists","up"]},{"osd":6,"uuid":"58ff5768-d850-40f2-b90c-1098dfdc3256","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":3834224571},{"type":"v1","addr":"172.21.15.60:6819","nonce":3834224571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":3834224571},{"type":"v1","addr":"172.21.15.60:6821","nonce":3834224571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":3834224571},{"type":"v1","addr":"172.21.15.60:6825","nonce":3834224571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":3834224571},{"type":"v1","addr":"172.21.15.60:6823","nonce":3834224571}]},"public_addr":"172.21.15.60:6819/3834224571","cluster_addr":"172.21.15.60:6821/3834224571","heartbeat_back_addr":"172.21.15.60:6825/3834224571","heartbeat_front_addr":"172.21.15.60:6823/3834224571","state":["exists","up"]},{"osd":7,"uuid":"375cd893-5f26-42bb-98ed-30b05911a7af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":404531062},{"type":"v1","addr":"172.21.15.136:6817","nonce":404531062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":404531062},{"type":"v1","addr":"172.21.15.136:6819","nonce":404531062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":404531062},{"type":"v1","addr":"172.21.15.136:6823","nonce":404531062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":404531062},{"type":"v1","addr":"172.21.15.136:6821","nonce":404531062}]},"public_addr":"172.21.15.136:6817/404531062","cluster_addr":"172.21.15.136:6819/404531062","heartbeat_back_addr":"172.21.15.136:6823/404531062","heartbeat_front_addr":"172.21.15.136:6821/404531062","state":["exists","up"]},{"osd":8,"uuid":"68b0051c-0fd3-4ea1-b7df-ea0b8409b8cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":502824503},{"type":"v1","addr":"172.21.15.183:6817","nonce":502824503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":502824503},{"type":"v1","addr":"172.21.15.183:6819","nonce":502824503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":502824503},{"type":"v1","addr":"172.21.15.183:6823","nonce":502824503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":502824503},{"type":"v1","addr":"172.21.15.183:6821","nonce":502824503}]},"public_addr":"172.21.15.183:6817/502824503","cluster_addr":"172.21.15.183:6819/502824503","heartbeat_back_addr":"172.21.15.183:6823/502824503","heartbeat_front_addr":"172.21.15.183:6821/502824503","state":["exists","up"]},{"osd":9,"uuid":"cafd4cc3-258c-4f88-8d3d-0eb4dfa2e2c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":634188302},{"type":"v1","addr":"172.21.15.60:6827","nonce":634188302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":634188302},{"type":"v1","addr":"172.21.15.60:6829","nonce":634188302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":634188302},{"type":"v1","addr":"172.21.15.60:6833","nonce":634188302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":634188302},{"type":"v1","addr":"172.21.15.60:6831","nonce":634188302}]},"public_addr":"172.21.15.60:6827/634188302","cluster_addr":"172.21.15.60:6829/634188302","heartbeat_back_addr":"172.21.15.60:6833/634188302","heartbeat_front_addr":"172.21.15.60:6831/634188302","state":["exists","up"]},{"osd":10,"uuid":"0b092b5a-8512-47d5-928b-995f8dc17353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":247446439},{"type":"v1","addr":"172.21.15.136:6825","nonce":247446439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":247446439},{"type":"v1","addr":"172.21.15.136:6827","nonce":247446439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":247446439},{"type":"v1","addr":"172.21.15.136:6831","nonce":247446439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":247446439},{"type":"v1","addr":"172.21.15.136:6829","nonce":247446439}]},"public_addr":"172.21.15.136:6825/247446439","cluster_addr":"172.21.15.136:6827/247446439","heartbeat_back_addr":"172.21.15.136:6831/247446439","heartbeat_front_addr":"172.21.15.136:6829/247446439","state":["exists","up"]},{"osd":11,"uuid":"8415b1a9-eb18-4b27-ad4f-9a676757bef9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1157339878},{"type":"v1","addr":"172.21.15.183:6825","nonce":1157339878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":1157339878},{"type":"v1","addr":"172.21.15.183:6827","nonce":1157339878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":1157339878},{"type":"v1","addr":"172.21.15.183:6831","nonce":1157339878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":1157339878},{"type":"v1","addr":"172.21.15.183:6829","nonce":1157339878}]},"public_addr":"172.21.15.183:6825/1157339878","cluster_addr":"172.21.15.183:6827/1157339878","heartbeat_back_addr":"172.21.15.183:6831/1157339878","heartbeat_front_addr":"172.21.15.183:6829/1157339878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:23:03.185905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:23:24.597431+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:23:45.809074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:24:06.953357+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:24:28.808398+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:24:50.196615+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:25:11.554360+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:25:32.969005+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:25:54.487750+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:26:15.703232+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:26:37.390344+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-20T01:26:58.959419+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2364926895":"2023-07-21T01:21:37.803443+0000","172.21.15.60:6800/712953724":"2023-07-21T01:21:37.803443+0000","172.21.15.60:0/3180624926":"2023-07-21T01:21:37.803443+0000","172.21.15.60:6801/712953724":"2023-07-21T01:21:37.803443+0000","172.21.15.60:0/1758646915":"2023-07-21T01:21:11.418252+0000","172.21.15.60:6801/1628989585":"2023-07-21T01:21:11.418252+0000","172.21.15.60:0/884024563":"2023-07-21T01:21:11.418252+0000","172.21.15.60:0/296900825":"2023-07-21T01:21:11.418252+0000","172.21.15.60:6800/1628989585":"2023-07-21T01:21:11.418252+0000","172.21.15.60:0/3556060174":"2023-07-21T01:21:37.803443+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-20T01:27:43.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.686+0000 7fcefeec9700 1 -- 172.21.15.60:0/2427076588 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fcee80a1220 msgr2=0x7fcee80a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:43.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.686+0000 7fcefeec9700 1 --2- 172.21.15.60:0/2427076588 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fcee80a1220 0x7fcee80a36e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fceec006010 tx=0x7fceec000bc0 comp rx=0 tx=0).stop 2023-07-20T01:27:43.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.686+0000 7fcefeec9700 1 -- 172.21.15.60:0/2427076588 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcef8101980 msgr2=0x7fcef8196950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:43.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.686+0000 7fcefeec9700 1 --2- 172.21.15.60:0/2427076588 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcef8101980 0x7fcef8196950 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fcee400b530 tx=0x7fcee400b840 comp rx=0 tx=0).stop 2023-07-20T01:27:43.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.686+0000 7fcefeec9700 1 -- 172.21.15.60:0/2427076588 shutdown_connections 2023-07-20T01:27:43.665 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.686+0000 7fcefeec9700 1 --2- 172.21.15.60:0/2427076588 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fcee80a1220 0x7fcee80a36e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:43.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.686+0000 7fcefeec9700 1 --2- 172.21.15.60:0/2427076588 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcef8102320 0x7fcef819d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:43.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.686+0000 7fcefeec9700 1 --2- 172.21.15.60:0/2427076588 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcef8100790 0x7fcef8196410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:43.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.686+0000 7fcefeec9700 1 --2- 172.21.15.60:0/2427076588 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcef8101980 0x7fcef8196950 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:43.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.686+0000 7fcefeec9700 1 -- 172.21.15.60:0/2427076588 >> 172.21.15.60:0/2427076588 conn(0x7fcef80fbd20 msgr2=0x7fcef80fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:43.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.686+0000 7fcefeec9700 1 -- 172.21.15.60:0/2427076588 shutdown_connections 2023-07-20T01:27:43.666 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.686+0000 7fcefeec9700 1 -- 172.21.15.60:0/2427076588 wait complete. 2023-07-20T01:27:43.677 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-07-20T01:27:43.808 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.831+0000 7f5304987700 1 -- 172.21.15.60:0/496000169 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5300101950 msgr2=0x7f5300101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:43.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.831+0000 7f5304987700 1 --2- 172.21.15.60:0/496000169 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5300101950 0x7f5300101db0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f52f40099c0 tx=0x7f52f4009cd0 comp rx=0 tx=0).stop 2023-07-20T01:27:43.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.831+0000 7f5304987700 1 -- 172.21.15.60:0/496000169 shutdown_connections 2023-07-20T01:27:43.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.831+0000 7f5304987700 1 --2- 172.21.15.60:0/496000169 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53001022f0 0x7f5300108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:43.809 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.831+0000 7f5304987700 1 --2- 172.21.15.60:0/496000169 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5300100760 0x7f5300100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:43.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.831+0000 7f5304987700 1 --2- 172.21.15.60:0/496000169 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5300101950 0x7f5300101db0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:43.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.831+0000 7f5304987700 1 -- 172.21.15.60:0/496000169 >> 172.21.15.60:0/496000169 conn(0x7f53000fbcf0 msgr2=0x7f53000fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:43.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.831+0000 7f5304987700 1 -- 172.21.15.60:0/496000169 shutdown_connections 2023-07-20T01:27:43.810 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.831+0000 7f5304987700 1 -- 172.21.15.60:0/496000169 wait complete. 2023-07-20T01:27:43.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.832+0000 7f5304987700 1 Processor -- start 2023-07-20T01:27:43.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.832+0000 7f5304987700 1 -- start start 2023-07-20T01:27:43.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.832+0000 7f5304987700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5300100760 0x7f53001962f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:43.812 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.832+0000 7f5304987700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5300101950 0x7f5300196830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:43.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.832+0000 7f5304987700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53001022f0 0x7f530019d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:43.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.832+0000 7f5304987700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530019de70 con 0x7f5300101950 2023-07-20T01:27:43.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.832+0000 7f5304987700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530019dfb0 con 0x7f53001022f0 2023-07-20T01:27:43.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.832+0000 7f5304987700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530019e140 con 0x7f5300100760 2023-07-20T01:27:43.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.832+0000 7f52fdd9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5300101950 0x7f5300196830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:43.813 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.832+0000 7f52fdd9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5300101950 0x7f5300196830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43162/0 (socket says 172.21.15.60:43162) 2023-07-20T01:27:43.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.832+0000 7f52fdd9b700 1 -- 172.21.15.60:0/2040072348 learned_addr learned my addr 172.21.15.60:0/2040072348 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:27:43.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.832+0000 7f52fed9d700 1 --2- 172.21.15.60:0/2040072348 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53001022f0 0x7f530019d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:43.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.832+0000 7f52fe59c700 1 --2- 172.21.15.60:0/2040072348 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5300100760 0x7f53001962f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:43.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.832+0000 7f52fdd9b700 1 -- 172.21.15.60:0/2040072348 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5300100760 msgr2=0x7f53001962f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:43.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.832+0000 7f52fdd9b700 1 --2- 172.21.15.60:0/2040072348 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5300100760 0x7f53001962f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:43.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.832+0000 7f52fdd9b700 1 -- 172.21.15.60:0/2040072348 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53001022f0 msgr2=0x7f530019d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:43.814 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.832+0000 7f52fdd9b700 1 --2- 172.21.15.60:0/2040072348 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53001022f0 0x7f530019d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:43.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.832+0000 7f52fdd9b700 1 -- 172.21.15.60:0/2040072348 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f40095f0 con 0x7f5300101950 2023-07-20T01:27:43.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.833+0000 7f52fdd9b700 1 --2- 172.21.15.60:0/2040072348 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5300101950 0x7f5300196830 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f52f4005910 tx=0x7f52f4003a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:43.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.833+0000 7f52ef7fe700 1 -- 172.21.15.60:0/2040072348 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f401d070 con 0x7f5300101950 2023-07-20T01:27:43.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.833+0000 7f5304987700 1 -- 172.21.15.60:0/2040072348 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f530019e3c0 con 0x7f5300101950 2023-07-20T01:27:43.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.833+0000 7f52ef7fe700 1 -- 172.21.15.60:0/2040072348 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f52f4022470 con 0x7f5300101950 2023-07-20T01:27:43.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.833+0000 7f52ef7fe700 1 -- 172.21.15.60:0/2040072348 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f400f6c0 con 0x7f5300101950 2023-07-20T01:27:43.815 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.833+0000 7f5304987700 1 -- 172.21.15.60:0/2040072348 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f530019e840 con 0x7f5300101950 2023-07-20T01:27:43.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.834+0000 7f5304987700 1 -- 172.21.15.60:0/2040072348 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53001076e0 con 0x7f5300101950 2023-07-20T01:27:43.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.835+0000 7f52ef7fe700 1 -- 172.21.15.60:0/2040072348 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f52f4022870 con 0x7f5300101950 2023-07-20T01:27:43.816 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.836+0000 7f52ef7fe700 1 --2- 172.21.15.60:0/2040072348 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f52e80a1150 0x7f52e80a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:43.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.836+0000 7f52fe59c700 1 --2- 172.21.15.60:0/2040072348 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f52e80a1150 0x7f52e80a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:43.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.836+0000 7f52ef7fe700 1 -- 172.21.15.60:0/2040072348 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f52f4022e40 con 0x7f5300101950 2023-07-20T01:27:43.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.836+0000 7f52fe59c700 1 --2- 172.21.15.60:0/2040072348 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f52e80a1150 0x7f52e80a3610 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f53001017b0 tx=0x7f52f000a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:43.817 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:43.840+0000 7f52ef7fe700 1 -- 172.21.15.60:0/2040072348 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f52f40976e0 con 0x7f5300101950 2023-07-20T01:27:44.067 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:44.089+0000 7f5304987700 1 -- 172.21.15.60:0/2040072348 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f530002c970 con 0x7f5300101950 2023-07-20T01:27:44.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:43 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/19132740' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-20T01:27:44.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:43 smithi060 ceph-mon[96524]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-20T01:27:44.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:43 smithi060 ceph-mon[96524]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-20T01:27:44.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:43 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:44.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:43 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:44.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:43 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:44.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:43 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:44.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:43 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/3569553728' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T01:27:44.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:43 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:44.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:43 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:44.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:43 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:44.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:43 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:27:44.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:43 smithi060 ceph-mon[96524]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:44.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:43 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/2427076588' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T01:27:44.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:43 smithi060 ceph-mon[96524]: osdmap e82: 12 total, 12 up, 12 in 2023-07-20T01:27:44.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:43 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/19132740' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-20T01:27:44.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:43 smithi183 ceph-mon[98477]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-20T01:27:44.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:43 smithi183 ceph-mon[98477]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-20T01:27:44.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:43 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:44.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:43 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:44.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:43 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:44.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:43 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:44.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:43 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/3569553728' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T01:27:44.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:43 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:44.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:43 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:44.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:43 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:44.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:43 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:27:44.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:43 smithi183 ceph-mon[98477]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:44.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:43 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/2427076588' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T01:27:44.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:43 smithi183 ceph-mon[98477]: osdmap e82: 12 total, 12 up, 12 in 2023-07-20T01:27:44.394 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:43 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/19132740' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-20T01:27:44.394 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:43 smithi136 ceph-mon[98083]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-20T01:27:44.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:43 smithi136 ceph-mon[98083]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-20T01:27:44.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:43 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:44.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:43 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:44.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:43 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:44.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:43 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:44.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:43 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/3569553728' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T01:27:44.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:43 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:44.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:43 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:44.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:43 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-20T01:27:44.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:43 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-20T01:27:44.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:43 smithi136 ceph-mon[98083]: from='mgr.14152 172.21.15.60:0/3766212965' entity='mgr.x' 2023-07-20T01:27:44.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:43 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/2427076588' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-20T01:27:44.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:43 smithi136 ceph-mon[98083]: osdmap e82: 12 total, 12 up, 12 in 2023-07-20T01:27:44.917 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:44.940+0000 7f52ef7fe700 1 -- 172.21.15.60:0/2040072348 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7f530002c970 con 0x7f5300101950 2023-07-20T01:27:44.920 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:44.943+0000 7f5304987700 1 -- 172.21.15.60:0/2040072348 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f52e80a1150 msgr2=0x7f52e80a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:44.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:44.943+0000 7f5304987700 1 --2- 172.21.15.60:0/2040072348 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f52e80a1150 0x7f52e80a3610 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f53001017b0 tx=0x7f52f000a3b0 comp rx=0 tx=0).stop 2023-07-20T01:27:44.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:44.943+0000 7f5304987700 1 -- 172.21.15.60:0/2040072348 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5300101950 msgr2=0x7f5300196830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:44.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:44.943+0000 7f5304987700 1 --2- 172.21.15.60:0/2040072348 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5300101950 0x7f5300196830 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f52f4005910 tx=0x7f52f4003a50 comp rx=0 tx=0).stop 2023-07-20T01:27:44.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:44.943+0000 7f5304987700 1 -- 172.21.15.60:0/2040072348 shutdown_connections 2023-07-20T01:27:44.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:44.943+0000 7f5304987700 1 --2- 172.21.15.60:0/2040072348 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f52e80a1150 0x7f52e80a3610 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:44.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:44.943+0000 7f5304987700 1 --2- 172.21.15.60:0/2040072348 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53001022f0 0x7f530019d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:44.921 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:44.943+0000 7f5304987700 1 --2- 172.21.15.60:0/2040072348 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5300100760 0x7f53001962f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:44.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:44.943+0000 7f5304987700 1 --2- 172.21.15.60:0/2040072348 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5300101950 0x7f5300196830 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:44.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:44.943+0000 7f5304987700 1 -- 172.21.15.60:0/2040072348 >> 172.21.15.60:0/2040072348 conn(0x7f53000fbcf0 msgr2=0x7f53000fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:44.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:44.943+0000 7f5304987700 1 -- 172.21.15.60:0/2040072348 shutdown_connections 2023-07-20T01:27:44.922 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:44.943+0000 7f5304987700 1 -- 172.21.15.60:0/2040072348 wait complete. 2023-07-20T01:27:44.935 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2023-07-20T01:27:44.936 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2023-07-20T01:27:45.075 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.094+0000 7f0d69b35700 1 -- 172.21.15.60:0/1836098096 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d64106460 msgr2=0x7f0d641088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:45.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.094+0000 7f0d69b35700 1 --2- 172.21.15.60:0/1836098096 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d64106460 0x7f0d641088d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c00b0a0 tx=0x7f0d5c00b3b0 comp rx=0 tx=0).stop 2023-07-20T01:27:45.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.094+0000 7f0d69b35700 1 -- 172.21.15.60:0/1836098096 shutdown_connections 2023-07-20T01:27:45.076 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.094+0000 7f0d69b35700 1 --2- 172.21.15.60:0/1836098096 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d64106460 0x7f0d641088d0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:45.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.094+0000 7f0d69b35700 1 --2- 172.21.15.60:0/1836098096 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d64101200 0x7f0d641035f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:45.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.094+0000 7f0d69b35700 1 --2- 172.21.15.60:0/1836098096 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d64103b30 0x7f0d64105f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:45.077 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.094+0000 7f0d69b35700 1 -- 172.21.15.60:0/1836098096 >> 172.21.15.60:0/1836098096 conn(0x7f0d640fac30 msgr2=0x7f0d640fd050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:45.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.094+0000 7f0d69b35700 1 -- 172.21.15.60:0/1836098096 shutdown_connections 2023-07-20T01:27:45.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.094+0000 7f0d69b35700 1 -- 172.21.15.60:0/1836098096 wait complete. 2023-07-20T01:27:45.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.095+0000 7f0d69b35700 1 Processor -- start 2023-07-20T01:27:45.078 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.095+0000 7f0d69b35700 1 -- start start 2023-07-20T01:27:45.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.095+0000 7f0d69b35700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d64101200 0x7f0d641964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:45.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.095+0000 7f0d69b35700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d64103b30 0x7f0d641969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:45.079 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.095+0000 7f0d69b35700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d64106460 0x7f0d6419da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:45.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.095+0000 7f0d69b35700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6419e030 con 0x7f0d64103b30 2023-07-20T01:27:45.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.095+0000 7f0d69b35700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6419e170 con 0x7f0d64101200 2023-07-20T01:27:45.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.095+0000 7f0d69b35700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6419e300 con 0x7f0d64106460 2023-07-20T01:27:45.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.095+0000 7f0d62ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d64103b30 0x7f0d641969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:45.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.095+0000 7f0d62ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d64103b30 0x7f0d641969f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43176/0 (socket says 172.21.15.60:43176) 2023-07-20T01:27:45.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.095+0000 7f0d62ffd700 1 -- 172.21.15.60:0/210686419 learned_addr learned my addr 172.21.15.60:0/210686419 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:27:45.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.095+0000 7f0d637fe700 1 --2- 172.21.15.60:0/210686419 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d64101200 0x7f0d641964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:45.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.095+0000 7f0d63fff700 1 --2- 172.21.15.60:0/210686419 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d64106460 0x7f0d6419da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:45.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.095+0000 7f0d62ffd700 1 -- 172.21.15.60:0/210686419 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d64106460 msgr2=0x7f0d6419da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:45.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.095+0000 7f0d62ffd700 1 --2- 172.21.15.60:0/210686419 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d64106460 0x7f0d6419da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:45.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.095+0000 7f0d62ffd700 1 -- 172.21.15.60:0/210686419 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d64101200 msgr2=0x7f0d641964b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:45.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.095+0000 7f0d62ffd700 1 --2- 172.21.15.60:0/210686419 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d64101200 0x7f0d641964b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:45.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.095+0000 7f0d62ffd700 1 -- 172.21.15.60:0/210686419 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d5c009d00 con 0x7f0d64103b30 2023-07-20T01:27:45.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.095+0000 7f0d63fff700 1 --2- 172.21.15.60:0/210686419 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d64106460 0x7f0d6419da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T01:27:45.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.096+0000 7f0d62ffd700 1 --2- 172.21.15.60:0/210686419 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d64103b30 0x7f0d641969f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0d5400d980 tx=0x7f0d5400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:45.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.096+0000 7f0d60ff9700 1 -- 172.21.15.60:0/210686419 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d54004220 con 0x7f0d64103b30 2023-07-20T01:27:45.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.096+0000 7f0d69b35700 1 -- 172.21.15.60:0/210686419 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d6419e5e0 con 0x7f0d64103b30 2023-07-20T01:27:45.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.096+0000 7f0d60ff9700 1 -- 172.21.15.60:0/210686419 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0d54007640 con 0x7f0d64103b30 2023-07-20T01:27:45.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.096+0000 7f0d60ff9700 1 -- 172.21.15.60:0/210686419 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d540055e0 con 0x7f0d64103b30 2023-07-20T01:27:45.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.096+0000 7f0d69b35700 1 -- 172.21.15.60:0/210686419 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d6419eaf0 con 0x7f0d64103b30 2023-07-20T01:27:45.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.097+0000 7f0d69b35700 1 -- 172.21.15.60:0/210686419 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d640fc7b0 con 0x7f0d64103b30 2023-07-20T01:27:45.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.098+0000 7f0d60ff9700 1 -- 172.21.15.60:0/210686419 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0d54004750 con 0x7f0d64103b30 2023-07-20T01:27:45.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.099+0000 7f0d60ff9700 1 --2- 172.21.15.60:0/210686419 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f0d500a1150 0x7f0d500a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:45.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.099+0000 7f0d637fe700 1 --2- 172.21.15.60:0/210686419 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f0d500a1150 0x7f0d500a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:45.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.099+0000 7f0d60ff9700 1 -- 172.21.15.60:0/210686419 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f0d540ccac0 con 0x7f0d64103b30 2023-07-20T01:27:45.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.099+0000 7f0d637fe700 1 --2- 172.21.15.60:0/210686419 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f0d500a1150 0x7f0d500a3610 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c005950 tx=0x7f0d4c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:45.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.102+0000 7f0d60ff9700 1 -- 172.21.15.60:0/210686419 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0d54096360 con 0x7f0d64103b30 2023-07-20T01:27:45.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:44 smithi060 ceph-mon[96524]: pgmap v255: 97 pgs: 10 creating+peering, 38 active+clean, 49 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 1 op/s 2023-07-20T01:27:45.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:44 smithi060 ceph-mon[96524]: mds.? [v2:172.21.15.136:6838/1778968612,v1:172.21.15.136:6839/1778968612] up:active 2023-07-20T01:27:45.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:44 smithi060 ceph-mon[96524]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T01:27:45.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:44 smithi060 ceph-mon[96524]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T01:27:45.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:44 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/2040072348' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-20T01:27:45.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:44 smithi183 ceph-mon[98477]: pgmap v255: 97 pgs: 10 creating+peering, 38 active+clean, 49 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 1 op/s 2023-07-20T01:27:45.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:44 smithi183 ceph-mon[98477]: mds.? [v2:172.21.15.136:6838/1778968612,v1:172.21.15.136:6839/1778968612] up:active 2023-07-20T01:27:45.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:44 smithi183 ceph-mon[98477]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T01:27:45.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:44 smithi183 ceph-mon[98477]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T01:27:45.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:44 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/2040072348' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-20T01:27:45.325 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.348+0000 7f0d69b35700 1 -- 172.21.15.60:0/210686419 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7f0d6402cbe0 con 0x7f0d64103b30 2023-07-20T01:27:45.394 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:44 smithi136 ceph-mon[98083]: pgmap v255: 97 pgs: 10 creating+peering, 38 active+clean, 49 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 1 op/s 2023-07-20T01:27:45.394 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:44 smithi136 ceph-mon[98083]: mds.? [v2:172.21.15.136:6838/1778968612,v1:172.21.15.136:6839/1778968612] up:active 2023-07-20T01:27:45.394 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:44 smithi136 ceph-mon[98083]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T01:27:45.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:44 smithi136 ceph-mon[98083]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T01:27:45.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:44 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/2040072348' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-20T01:27:45.923 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.945+0000 7f0d60ff9700 1 -- 172.21.15.60:0/210686419 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v18) v1 ==== 105+0+0 (secure 0 0 0) 0x7f0d54016020 con 0x7f0d64103b30 2023-07-20T01:27:45.925 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.948+0000 7f0d69b35700 1 -- 172.21.15.60:0/210686419 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f0d500a1150 msgr2=0x7f0d500a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:45.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.948+0000 7f0d69b35700 1 --2- 172.21.15.60:0/210686419 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f0d500a1150 0x7f0d500a3610 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c005950 tx=0x7f0d4c00b410 comp rx=0 tx=0).stop 2023-07-20T01:27:45.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.948+0000 7f0d69b35700 1 -- 172.21.15.60:0/210686419 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d64103b30 msgr2=0x7f0d641969f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:45.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.948+0000 7f0d69b35700 1 --2- 172.21.15.60:0/210686419 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d64103b30 0x7f0d641969f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0d5400d980 tx=0x7f0d5400dc90 comp rx=0 tx=0).stop 2023-07-20T01:27:45.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.948+0000 7f0d69b35700 1 -- 172.21.15.60:0/210686419 shutdown_connections 2023-07-20T01:27:45.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.948+0000 7f0d69b35700 1 --2- 172.21.15.60:0/210686419 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f0d500a1150 0x7f0d500a3610 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:45.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.948+0000 7f0d69b35700 1 --2- 172.21.15.60:0/210686419 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d64106460 0x7f0d6419da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:45.926 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.948+0000 7f0d69b35700 1 --2- 172.21.15.60:0/210686419 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d64101200 0x7f0d641964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:45.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.948+0000 7f0d69b35700 1 --2- 172.21.15.60:0/210686419 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d64103b30 0x7f0d641969f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:45.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.948+0000 7f0d69b35700 1 -- 172.21.15.60:0/210686419 >> 172.21.15.60:0/210686419 conn(0x7f0d640fac30 msgr2=0x7f0d64101e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:45.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.948+0000 7f0d69b35700 1 -- 172.21.15.60:0/210686419 shutdown_connections 2023-07-20T01:27:45.927 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:45.948+0000 7f0d69b35700 1 -- 172.21.15.60:0/210686419 wait complete. 2023-07-20T01:27:45.943 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2023-07-20T01:27:46.080 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.102+0000 7fae6ddf0700 1 -- 172.21.15.60:0/3730623623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae68100c60 msgr2=0x7fae68101080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:46.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.102+0000 7fae6ddf0700 1 --2- 172.21.15.60:0/3730623623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae68100c60 0x7fae68101080 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fae58009990 tx=0x7fae58009ca0 comp rx=0 tx=0).stop 2023-07-20T01:27:46.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.102+0000 7fae6ddf0700 1 -- 172.21.15.60:0/3730623623 shutdown_connections 2023-07-20T01:27:46.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.102+0000 7fae6ddf0700 1 --2- 172.21.15.60:0/3730623623 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae68100270 0x7fae68107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:46.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.102+0000 7fae6ddf0700 1 --2- 172.21.15.60:0/3730623623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae68100c60 0x7fae68101080 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:46.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.102+0000 7fae6ddf0700 1 --2- 172.21.15.60:0/3730623623 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae680ff8b0 0x7fae680ffd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:46.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.102+0000 7fae6ddf0700 1 -- 172.21.15.60:0/3730623623 >> 172.21.15.60:0/3730623623 conn(0x7fae680fa340 msgr2=0x7fae680fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:46.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.102+0000 7fae6ddf0700 1 -- 172.21.15.60:0/3730623623 shutdown_connections 2023-07-20T01:27:46.081 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.102+0000 7fae6ddf0700 1 -- 172.21.15.60:0/3730623623 wait complete. 2023-07-20T01:27:46.082 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.103+0000 7fae6ddf0700 1 Processor -- start 2023-07-20T01:27:46.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.103+0000 7fae6ddf0700 1 -- start start 2023-07-20T01:27:46.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.103+0000 7fae6ddf0700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae680ff8b0 0x7fae680ff370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:46.083 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.103+0000 7fae6ddf0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae68100270 0x7fae6819b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:46.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.103+0000 7fae6ddf0700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae68100c60 0x7fae6819d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:46.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.103+0000 7fae6ddf0700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6819df00 con 0x7fae68100270 2023-07-20T01:27:46.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.103+0000 7fae6ddf0700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6819e070 con 0x7fae68100c60 2023-07-20T01:27:46.084 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.103+0000 7fae6ddf0700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6819e230 con 0x7fae680ff8b0 2023-07-20T01:27:46.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.103+0000 7fae66ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae68100270 0x7fae6819b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:46.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.103+0000 7fae66ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae68100270 0x7fae6819b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43190/0 (socket says 172.21.15.60:43190) 2023-07-20T01:27:46.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.103+0000 7fae66ffd700 1 -- 172.21.15.60:0/668377103 learned_addr learned my addr 172.21.15.60:0/668377103 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:27:46.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.103+0000 7fae677fe700 1 --2- 172.21.15.60:0/668377103 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae680ff8b0 0x7fae680ff370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:46.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.104+0000 7fae67fff700 1 --2- 172.21.15.60:0/668377103 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae68100c60 0x7fae6819d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:46.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.104+0000 7fae66ffd700 1 -- 172.21.15.60:0/668377103 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae680ff8b0 msgr2=0x7fae680ff370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:46.085 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.104+0000 7fae66ffd700 1 --2- 172.21.15.60:0/668377103 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae680ff8b0 0x7fae680ff370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:46.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.104+0000 7fae66ffd700 1 -- 172.21.15.60:0/668377103 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae68100c60 msgr2=0x7fae6819d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:46.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.104+0000 7fae66ffd700 1 --2- 172.21.15.60:0/668377103 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae68100c60 0x7fae6819d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:46.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.104+0000 7fae66ffd700 1 -- 172.21.15.60:0/668377103 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae580095f0 con 0x7fae68100270 2023-07-20T01:27:46.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.104+0000 7fae67fff700 1 --2- 172.21.15.60:0/668377103 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae68100c60 0x7fae6819d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-20T01:27:46.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.104+0000 7fae66ffd700 1 --2- 172.21.15.60:0/668377103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae68100270 0x7fae6819b0d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fae5c00c920 tx=0x7fae5c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:46.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.104+0000 7fae64ff9700 1 -- 172.21.15.60:0/668377103 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5c007b00 con 0x7fae68100270 2023-07-20T01:27:46.086 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.104+0000 7fae64ff9700 1 -- 172.21.15.60:0/668377103 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fae5c00ce70 con 0x7fae68100270 2023-07-20T01:27:46.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.104+0000 7fae6ddf0700 1 -- 172.21.15.60:0/668377103 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae6819e510 con 0x7fae68100270 2023-07-20T01:27:46.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.104+0000 7fae64ff9700 1 -- 172.21.15.60:0/668377103 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5c005440 con 0x7fae68100270 2023-07-20T01:27:46.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.104+0000 7fae6ddf0700 1 -- 172.21.15.60:0/668377103 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae6819ea20 con 0x7fae68100270 2023-07-20T01:27:46.087 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.105+0000 7fae6ddf0700 1 -- 172.21.15.60:0/668377103 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae68106740 con 0x7fae68100270 2023-07-20T01:27:46.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.106+0000 7fae64ff9700 1 -- 172.21.15.60:0/668377103 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fae5c0055a0 con 0x7fae68100270 2023-07-20T01:27:46.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.107+0000 7fae64ff9700 1 --2- 172.21.15.60:0/668377103 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fae500a1150 0x7fae500a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:46.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.107+0000 7fae677fe700 1 --2- 172.21.15.60:0/668377103 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fae500a1150 0x7fae500a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:46.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.107+0000 7fae64ff9700 1 -- 172.21.15.60:0/668377103 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fae5c0cc800 con 0x7fae68100270 2023-07-20T01:27:46.088 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.107+0000 7fae677fe700 1 --2- 172.21.15.60:0/668377103 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fae500a1150 0x7fae500a3610 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fae5800b5c0 tx=0x7fae58005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:46.089 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.110+0000 7fae64ff9700 1 -- 172.21.15.60:0/668377103 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fae5c096000 con 0x7fae68100270 2023-07-20T01:27:46.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:45 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/2040072348' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-20T01:27:46.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:45 smithi060 ceph-mon[96524]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T01:27:46.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:45 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/210686419' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2023-07-20T01:27:46.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:45 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/2040072348' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-20T01:27:46.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:45 smithi183 ceph-mon[98477]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T01:27:46.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:45 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/210686419' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2023-07-20T01:27:46.333 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.356+0000 7fae6ddf0700 1 -- 172.21.15.60:0/668377103 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7fae6802c970 con 0x7fae68100270 2023-07-20T01:27:46.394 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:45 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/2040072348' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-20T01:27:46.394 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:45 smithi136 ceph-mon[98083]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T01:27:46.394 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:45 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/210686419' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2023-07-20T01:27:46.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.712+0000 7fae64ff9700 1 -- 172.21.15.60:0/668377103 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) v1 ==== 121+0+0 (secure 0 0 0) 0x7fae5c01d020 con 0x7fae68100270 2023-07-20T01:27:46.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.715+0000 7fae6ddf0700 1 -- 172.21.15.60:0/668377103 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fae500a1150 msgr2=0x7fae500a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:46.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.715+0000 7fae6ddf0700 1 --2- 172.21.15.60:0/668377103 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fae500a1150 0x7fae500a3610 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fae5800b5c0 tx=0x7fae58005fb0 comp rx=0 tx=0).stop 2023-07-20T01:27:46.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.715+0000 7fae6ddf0700 1 -- 172.21.15.60:0/668377103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae68100270 msgr2=0x7fae6819b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:46.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.715+0000 7fae6ddf0700 1 --2- 172.21.15.60:0/668377103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae68100270 0x7fae6819b0d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fae5c00c920 tx=0x7fae5c00cc30 comp rx=0 tx=0).stop 2023-07-20T01:27:46.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.715+0000 7fae6ddf0700 1 -- 172.21.15.60:0/668377103 shutdown_connections 2023-07-20T01:27:46.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.715+0000 7fae6ddf0700 1 --2- 172.21.15.60:0/668377103 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7fae500a1150 0x7fae500a3610 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:46.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.715+0000 7fae6ddf0700 1 --2- 172.21.15.60:0/668377103 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae68100c60 0x7fae6819d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:46.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.715+0000 7fae6ddf0700 1 --2- 172.21.15.60:0/668377103 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae680ff8b0 0x7fae680ff370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:46.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.715+0000 7fae6ddf0700 1 --2- 172.21.15.60:0/668377103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae68100270 0x7fae6819b0d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:46.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.715+0000 7fae6ddf0700 1 -- 172.21.15.60:0/668377103 >> 172.21.15.60:0/668377103 conn(0x7fae680fa340 msgr2=0x7fae680fc6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:46.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.715+0000 7fae6ddf0700 1 -- 172.21.15.60:0/668377103 shutdown_connections 2023-07-20T01:27:46.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.715+0000 7fae6ddf0700 1 -- 172.21.15.60:0/668377103 wait complete. 2023-07-20T01:27:46.709 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-07-20T01:27:46.839 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.862+0000 7f4ff22a9700 1 -- 172.21.15.60:0/247962470 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fec1011f0 msgr2=0x7f4fec107aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:46.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.862+0000 7f4ff22a9700 1 --2- 172.21.15.60:0/247962470 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fec1011f0 0x7f4fec107aa0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4fe400b0d0 tx=0x7f4fe400b3e0 comp rx=0 tx=0).stop 2023-07-20T01:27:46.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.862+0000 7f4ff22a9700 1 -- 172.21.15.60:0/247962470 shutdown_connections 2023-07-20T01:27:46.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.862+0000 7f4ff22a9700 1 --2- 172.21.15.60:0/247962470 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fec1011f0 0x7f4fec107aa0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:46.840 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.862+0000 7f4ff22a9700 1 --2- 172.21.15.60:0/247962470 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4fec0ff660 0x7f4fec0ffa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:46.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.862+0000 7f4ff22a9700 1 --2- 172.21.15.60:0/247962470 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4fec100850 0x7f4fec100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:46.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.862+0000 7f4ff22a9700 1 -- 172.21.15.60:0/247962470 >> 172.21.15.60:0/247962470 conn(0x7f4fec0fabd0 msgr2=0x7f4fec0fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:46.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.862+0000 7f4ff22a9700 1 -- 172.21.15.60:0/247962470 shutdown_connections 2023-07-20T01:27:46.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.862+0000 7f4ff22a9700 1 -- 172.21.15.60:0/247962470 wait complete. 2023-07-20T01:27:46.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.862+0000 7f4ff22a9700 1 Processor -- start 2023-07-20T01:27:46.841 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.862+0000 7f4ff22a9700 1 -- start start 2023-07-20T01:27:46.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.863+0000 7f4ff22a9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fec0ff660 0x7f4fec068440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:46.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.863+0000 7f4ff22a9700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4fec100850 0x7f4fec19b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:46.843 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.863+0000 7f4ff22a9700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4fec1011f0 0x7f4fec19d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:46.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.863+0000 7f4ff22a9700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec19dfe0 con 0x7f4fec0ff660 2023-07-20T01:27:46.844 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.863+0000 7f4ff22a9700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec19e150 con 0x7f4fec1011f0 2023-07-20T01:27:46.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.863+0000 7f4ff22a9700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec19e310 con 0x7f4fec100850 2023-07-20T01:27:46.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.863+0000 7f4febfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fec0ff660 0x7f4fec068440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:46.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.863+0000 7f4febfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fec0ff660 0x7f4fec068440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43202/0 (socket says 172.21.15.60:43202) 2023-07-20T01:27:46.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.863+0000 7f4febfff700 1 -- 172.21.15.60:0/459039830 learned_addr learned my addr 172.21.15.60:0/459039830 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:27:46.845 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.863+0000 7f4feb7fe700 1 --2- 172.21.15.60:0/459039830 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4fec100850 0x7f4fec19b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:46.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.863+0000 7f4febfff700 1 -- 172.21.15.60:0/459039830 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4fec100850 msgr2=0x7f4fec19b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:46.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.863+0000 7f4febfff700 1 --2- 172.21.15.60:0/459039830 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4fec100850 0x7f4fec19b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:46.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.863+0000 7f4febfff700 1 -- 172.21.15.60:0/459039830 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4fec1011f0 msgr2=0x7f4fec19d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:46.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.863+0000 7f4febfff700 1 --2- 172.21.15.60:0/459039830 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4fec1011f0 0x7f4fec19d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:46.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.863+0000 7f4febfff700 1 -- 172.21.15.60:0/459039830 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe4009d00 con 0x7f4fec0ff660 2023-07-20T01:27:46.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.863+0000 7f4feb7fe700 1 --2- 172.21.15.60:0/459039830 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4fec100850 0x7f4fec19b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-20T01:27:46.846 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.863+0000 7f4febfff700 1 --2- 172.21.15.60:0/459039830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fec0ff660 0x7f4fec068440 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc00d980 tx=0x7f4fdc00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:46.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.863+0000 7f4fe97fa700 1 -- 172.21.15.60:0/459039830 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc00de60 con 0x7f4fec0ff660 2023-07-20T01:27:46.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.863+0000 7f4ff22a9700 1 -- 172.21.15.60:0/459039830 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fec19e5f0 con 0x7f4fec0ff660 2023-07-20T01:27:46.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.863+0000 7f4fe97fa700 1 -- 172.21.15.60:0/459039830 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4fdc0044b0 con 0x7f4fec0ff660 2023-07-20T01:27:46.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.863+0000 7f4fe97fa700 1 -- 172.21.15.60:0/459039830 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc003b40 con 0x7f4fec0ff660 2023-07-20T01:27:46.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.863+0000 7f4ff22a9700 1 -- 172.21.15.60:0/459039830 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fec19ebd0 con 0x7f4fec0ff660 2023-07-20T01:27:46.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.864+0000 7f4ff22a9700 1 -- 172.21.15.60:0/459039830 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fec1065e0 con 0x7f4fec0ff660 2023-07-20T01:27:46.847 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.866+0000 7f4fe97fa700 1 -- 172.21.15.60:0/459039830 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4fdc004870 con 0x7f4fec0ff660 2023-07-20T01:27:46.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.867+0000 7f4fe97fa700 1 --2- 172.21.15.60:0/459039830 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f4fd80a1150 0x7f4fd80a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:46.848 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.867+0000 7f4feb7fe700 1 --2- 172.21.15.60:0/459039830 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f4fd80a1150 0x7f4fd80a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:46.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.867+0000 7f4fe97fa700 1 -- 172.21.15.60:0/459039830 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f4fdc0cc910 con 0x7f4fec0ff660 2023-07-20T01:27:46.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.867+0000 7f4feb7fe700 1 --2- 172.21.15.60:0/459039830 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f4fd80a1150 0x7f4fd80a3610 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4fd4007900 tx=0x7f4fd4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:46.849 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:46.870+0000 7f4fe97fa700 1 -- 172.21.15.60:0/459039830 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4fdc096110 con 0x7f4fec0ff660 2023-07-20T01:27:47.100 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.122+0000 7f4ff22a9700 1 -- 172.21.15.60:0/459039830 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f4fec02cc20 con 0x7f4fec0ff660 2023-07-20T01:27:47.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:46 smithi060 ceph-mon[96524]: pgmap v256: 97 pgs: 10 creating+peering, 69 active+clean, 18 unknown; 578 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 3 op/s 2023-07-20T01:27:47.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:46 smithi060 ceph-mon[96524]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-20T01:27:47.174 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:46 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/210686419' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2023-07-20T01:27:47.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:46 smithi060 ceph-mon[96524]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-20T01:27:47.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:46 smithi060 ceph-mon[96524]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T01:27:47.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:46 smithi060 ceph-mon[96524]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2023-07-20T01:27:47.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:46 smithi060 ceph-mon[96524]: daemon mds.h is now active in filesystem cephfs as rank 1 2023-07-20T01:27:47.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:46 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/668377103' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-20T01:27:47.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:46 smithi060 ceph-mon[96524]: mds.? [v2:172.21.15.136:6836/895972199,v1:172.21.15.136:6837/895972199] up:active 2023-07-20T01:27:47.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:46 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/668377103' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-20T01:27:47.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:46 smithi060 ceph-mon[96524]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-20T01:27:47.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:46 smithi060 ceph-mon[96524]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 2 up:standby-replay 8 up:standby 2023-07-20T01:27:47.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:46 smithi060 ceph-mon[96524]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 2 up:standby-replay 7 up:standby 2023-07-20T01:27:47.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:46 smithi060 ceph-mon[96524]: daemon mds.e is now active in filesystem cephfs as rank 2 2023-07-20T01:27:47.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:46 smithi183 ceph-mon[98477]: pgmap v256: 97 pgs: 10 creating+peering, 69 active+clean, 18 unknown; 578 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 3 op/s 2023-07-20T01:27:47.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:46 smithi183 ceph-mon[98477]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-20T01:27:47.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:46 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/210686419' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2023-07-20T01:27:47.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:46 smithi183 ceph-mon[98477]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-20T01:27:47.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:46 smithi183 ceph-mon[98477]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T01:27:47.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:46 smithi183 ceph-mon[98477]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2023-07-20T01:27:47.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:46 smithi183 ceph-mon[98477]: daemon mds.h is now active in filesystem cephfs as rank 1 2023-07-20T01:27:47.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:46 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/668377103' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-20T01:27:47.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:46 smithi183 ceph-mon[98477]: mds.? [v2:172.21.15.136:6836/895972199,v1:172.21.15.136:6837/895972199] up:active 2023-07-20T01:27:47.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:46 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/668377103' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-20T01:27:47.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:46 smithi183 ceph-mon[98477]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-20T01:27:47.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:46 smithi183 ceph-mon[98477]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 2 up:standby-replay 8 up:standby 2023-07-20T01:27:47.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:46 smithi183 ceph-mon[98477]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 2 up:standby-replay 7 up:standby 2023-07-20T01:27:47.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:46 smithi183 ceph-mon[98477]: daemon mds.e is now active in filesystem cephfs as rank 2 2023-07-20T01:27:47.394 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:46 smithi136 ceph-mon[98083]: pgmap v256: 97 pgs: 10 creating+peering, 69 active+clean, 18 unknown; 578 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 3 op/s 2023-07-20T01:27:47.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:46 smithi136 ceph-mon[98083]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-20T01:27:47.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:46 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/210686419' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2023-07-20T01:27:47.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:46 smithi136 ceph-mon[98083]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-20T01:27:47.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:46 smithi136 ceph-mon[98083]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-20T01:27:47.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:46 smithi136 ceph-mon[98083]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2023-07-20T01:27:47.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:46 smithi136 ceph-mon[98083]: daemon mds.h is now active in filesystem cephfs as rank 1 2023-07-20T01:27:47.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:46 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/668377103' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-20T01:27:47.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:46 smithi136 ceph-mon[98083]: mds.? [v2:172.21.15.136:6836/895972199,v1:172.21.15.136:6837/895972199] up:active 2023-07-20T01:27:47.396 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:46 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/668377103' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-20T01:27:47.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:46 smithi136 ceph-mon[98083]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-20T01:27:47.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:46 smithi136 ceph-mon[98083]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 2 up:standby-replay 8 up:standby 2023-07-20T01:27:47.397 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:46 smithi136 ceph-mon[98083]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 2 up:standby-replay 7 up:standby 2023-07-20T01:27:47.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:46 smithi136 ceph-mon[98083]: daemon mds.e is now active in filesystem cephfs as rank 2 2023-07-20T01:27:47.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.719+0000 7f4fe97fa700 1 -- 172.21.15.60:0/459039830 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) v1 ==== 115+0+0 (secure 0 0 0) 0x7f4fdc009940 con 0x7f4fec0ff660 2023-07-20T01:27:47.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.722+0000 7f4ff22a9700 1 -- 172.21.15.60:0/459039830 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f4fd80a1150 msgr2=0x7f4fd80a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:47.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.722+0000 7f4ff22a9700 1 --2- 172.21.15.60:0/459039830 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f4fd80a1150 0x7f4fd80a3610 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4fd4007900 tx=0x7f4fd4008040 comp rx=0 tx=0).stop 2023-07-20T01:27:47.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.722+0000 7f4ff22a9700 1 -- 172.21.15.60:0/459039830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fec0ff660 msgr2=0x7f4fec068440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:47.700 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.722+0000 7f4ff22a9700 1 --2- 172.21.15.60:0/459039830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fec0ff660 0x7f4fec068440 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc00d980 tx=0x7f4fdc00dd40 comp rx=0 tx=0).stop 2023-07-20T01:27:47.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.722+0000 7f4ff22a9700 1 -- 172.21.15.60:0/459039830 shutdown_connections 2023-07-20T01:27:47.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.722+0000 7f4ff22a9700 1 --2- 172.21.15.60:0/459039830 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f4fd80a1150 0x7f4fd80a3610 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:47.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.722+0000 7f4ff22a9700 1 --2- 172.21.15.60:0/459039830 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4fec1011f0 0x7f4fec19d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:47.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.723+0000 7f4ff22a9700 1 --2- 172.21.15.60:0/459039830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fec0ff660 0x7f4fec068440 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:47.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.723+0000 7f4ff22a9700 1 --2- 172.21.15.60:0/459039830 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4fec100850 0x7f4fec19b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:47.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.723+0000 7f4ff22a9700 1 -- 172.21.15.60:0/459039830 >> 172.21.15.60:0/459039830 conn(0x7f4fec0fabd0 msgr2=0x7f4fec0fcec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:47.701 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.723+0000 7f4ff22a9700 1 -- 172.21.15.60:0/459039830 shutdown_connections 2023-07-20T01:27:47.702 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.723+0000 7f4ff22a9700 1 -- 172.21.15.60:0/459039830 wait complete. 2023-07-20T01:27:47.719 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2023-07-20T01:27:47.719 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2023-07-20T01:27:47.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.878+0000 7f5598350700 1 -- 172.21.15.60:0/2864826334 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55900ff910 msgr2=0x7f55900ffd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:47.855 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.878+0000 7f5598350700 1 --2- 172.21.15.60:0/2864826334 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55900ff910 0x7f55900ffd70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f558c009990 tx=0x7f558c009ca0 comp rx=0 tx=0).stop 2023-07-20T01:27:47.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.878+0000 7f5598350700 1 -- 172.21.15.60:0/2864826334 shutdown_connections 2023-07-20T01:27:47.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.878+0000 7f5598350700 1 --2- 172.21.15.60:0/2864826334 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f55901002b0 0x7f5590107b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:47.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.878+0000 7f5598350700 1 --2- 172.21.15.60:0/2864826334 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5590100be0 0x7f5590101000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:47.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.878+0000 7f5598350700 1 --2- 172.21.15.60:0/2864826334 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55900ff910 0x7f55900ffd70 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:47.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.878+0000 7f5598350700 1 -- 172.21.15.60:0/2864826334 >> 172.21.15.60:0/2864826334 conn(0x7f55900fa340 msgr2=0x7f55900fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:47.856 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.878+0000 7f5598350700 1 -- 172.21.15.60:0/2864826334 shutdown_connections 2023-07-20T01:27:47.857 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.878+0000 7f5598350700 1 -- 172.21.15.60:0/2864826334 wait complete. 2023-07-20T01:27:47.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.878+0000 7f5598350700 1 Processor -- start 2023-07-20T01:27:47.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.878+0000 7f5598350700 1 -- start start 2023-07-20T01:27:47.858 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.878+0000 7f5598350700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f55900ff910 0x7f55900ff370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:47.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.879+0000 7f5598350700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55901002b0 0x7f559019b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:47.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.879+0000 7f5598350700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5590100be0 0x7f559019d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:47.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.879+0000 7f5598350700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559019ded0 con 0x7f55901002b0 2023-07-20T01:27:47.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.879+0000 7f5598350700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559019e040 con 0x7f55900ff910 2023-07-20T01:27:47.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.879+0000 7f5598350700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559019e200 con 0x7f5590100be0 2023-07-20T01:27:47.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.879+0000 7f55958eb700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55901002b0 0x7f559019b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:47.859 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.879+0000 7f55958eb700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55901002b0 0x7f559019b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43218/0 (socket says 172.21.15.60:43218) 2023-07-20T01:27:47.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.879+0000 7f55958eb700 1 -- 172.21.15.60:0/1719717878 learned_addr learned my addr 172.21.15.60:0/1719717878 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:27:47.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.879+0000 7f55960ec700 1 --2- 172.21.15.60:0/1719717878 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f55900ff910 0x7f55900ff370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:47.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.879+0000 7f55968ed700 1 --2- 172.21.15.60:0/1719717878 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5590100be0 0x7f559019d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:47.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.879+0000 7f55958eb700 1 -- 172.21.15.60:0/1719717878 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5590100be0 msgr2=0x7f559019d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:47.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.879+0000 7f55958eb700 1 --2- 172.21.15.60:0/1719717878 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5590100be0 0x7f559019d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:47.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.879+0000 7f55958eb700 1 -- 172.21.15.60:0/1719717878 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f55900ff910 msgr2=0x7f55900ff370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:47.860 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.879+0000 7f55958eb700 1 --2- 172.21.15.60:0/1719717878 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f55900ff910 0x7f55900ff370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:47.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.879+0000 7f55958eb700 1 -- 172.21.15.60:0/1719717878 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f558c0095f0 con 0x7f55901002b0 2023-07-20T01:27:47.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.879+0000 7f55958eb700 1 --2- 172.21.15.60:0/1719717878 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55901002b0 0x7f559019b0f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f558c005910 tx=0x7f558c005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:47.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.879+0000 7f55877fe700 1 -- 172.21.15.60:0/1719717878 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558c01d070 con 0x7f55901002b0 2023-07-20T01:27:47.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.879+0000 7f55877fe700 1 -- 172.21.15.60:0/1719717878 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f558c004040 con 0x7f55901002b0 2023-07-20T01:27:47.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.879+0000 7f5598350700 1 -- 172.21.15.60:0/1719717878 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f559019e480 con 0x7f55901002b0 2023-07-20T01:27:47.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.879+0000 7f55877fe700 1 -- 172.21.15.60:0/1719717878 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558c005290 con 0x7f55901002b0 2023-07-20T01:27:47.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.879+0000 7f5598350700 1 -- 172.21.15.60:0/1719717878 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f559019e900 con 0x7f55901002b0 2023-07-20T01:27:47.861 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.880+0000 7f5598350700 1 -- 172.21.15.60:0/1719717878 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f559010ad40 con 0x7f55901002b0 2023-07-20T01:27:47.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.881+0000 7f55877fe700 1 -- 172.21.15.60:0/1719717878 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f558c0047e0 con 0x7f55901002b0 2023-07-20T01:27:47.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.882+0000 7f55877fe700 1 --2- 172.21.15.60:0/1719717878 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f557c0a11d0 0x7f557c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:47.862 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.882+0000 7f55960ec700 1 --2- 172.21.15.60:0/1719717878 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f557c0a11d0 0x7f557c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:47.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.882+0000 7f55877fe700 1 -- 172.21.15.60:0/1719717878 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f558c0cdc70 con 0x7f55901002b0 2023-07-20T01:27:47.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.882+0000 7f55960ec700 1 --2- 172.21.15.60:0/1719717878 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f557c0a11d0 0x7f557c0a3690 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5580005950 tx=0x7f55800058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:47.863 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:47.885+0000 7f55877fe700 1 -- 172.21.15.60:0/1719717878 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f558c097500 con 0x7f55901002b0 2023-07-20T01:27:47.894 INFO:journalctl@ceph.mds.b.smithi136.stdout:Jul 20 01:27:47 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-b[123171]: 2023-07-20T01:27:47.514+0000 7ff2ba8e5700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-07-20T01:27:47.894 INFO:journalctl@ceph.mds.b.smithi136.stdout:Jul 20 01:27:47 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-b[123171]: 2023-07-20T01:27:47.514+0000 7ff2ba8e5700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-07-20T01:27:48.094 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.116+0000 7f5598350700 1 -- 172.21.15.60:0/1719717878 --> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f5590060bb0 con 0x7f557c0a11d0 2023-07-20T01:27:48.101 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:47 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/459039830' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-20T01:27:48.101 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:47 smithi060 ceph-mon[96524]: mds.? [v2:172.21.15.136:6834/3349527335,v1:172.21.15.136:6835/3349527335] up:active 2023-07-20T01:27:48.101 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:47 smithi060 ceph-mon[96524]: from='client.? 172.21.15.60:0/459039830' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-20T01:27:48.101 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:47 smithi060 ceph-mon[96524]: daemon mds.f assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2023-07-20T01:27:48.101 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:47 smithi060 ceph-mon[96524]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2023-07-20T01:27:48.102 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:47 smithi060 ceph-mon[96524]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=f=up:creating} 2 up:standby-replay 6 up:standby 2023-07-20T01:27:48.102 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:47 smithi060 ceph-mon[96524]: daemon mds.f is now active in filesystem cephfs as rank 3 2023-07-20T01:27:48.121 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.144+0000 7f55877fe700 1 -- 172.21.15.60:0/1719717878 <== mgr.14152 v2:172.21.15.60:6800/1961833180 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5590060bb0 con 0x7f557c0a11d0 2023-07-20T01:27:48.123 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.146+0000 7f5598350700 1 -- 172.21.15.60:0/1719717878 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f557c0a11d0 msgr2=0x7f557c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:48.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.146+0000 7f5598350700 1 --2- 172.21.15.60:0/1719717878 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f557c0a11d0 0x7f557c0a3690 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5580005950 tx=0x7f55800058e0 comp rx=0 tx=0).stop 2023-07-20T01:27:48.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.146+0000 7f5598350700 1 -- 172.21.15.60:0/1719717878 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55901002b0 msgr2=0x7f559019b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:48.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.146+0000 7f5598350700 1 --2- 172.21.15.60:0/1719717878 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55901002b0 0x7f559019b0f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f558c005910 tx=0x7f558c005ef0 comp rx=0 tx=0).stop 2023-07-20T01:27:48.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.146+0000 7f5598350700 1 -- 172.21.15.60:0/1719717878 shutdown_connections 2023-07-20T01:27:48.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.146+0000 7f5598350700 1 --2- 172.21.15.60:0/1719717878 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f557c0a11d0 0x7f557c0a3690 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:48.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.146+0000 7f5598350700 1 --2- 172.21.15.60:0/1719717878 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5590100be0 0x7f559019d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:48.124 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.146+0000 7f5598350700 1 --2- 172.21.15.60:0/1719717878 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f55900ff910 0x7f55900ff370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:48.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.146+0000 7f5598350700 1 --2- 172.21.15.60:0/1719717878 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55901002b0 0x7f559019b0f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:48.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.146+0000 7f5598350700 1 -- 172.21.15.60:0/1719717878 >> 172.21.15.60:0/1719717878 conn(0x7f55900fa340 msgr2=0x7f55900fc6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:48.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.146+0000 7f5598350700 1 -- 172.21.15.60:0/1719717878 shutdown_connections 2023-07-20T01:27:48.125 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.146+0000 7f5598350700 1 -- 172.21.15.60:0/1719717878 wait complete. 2023-07-20T01:27:48.136 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs --group_name=qa sv_0 '' 2023-07-20T01:27:48.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:47 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/459039830' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-20T01:27:48.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:47 smithi183 ceph-mon[98477]: mds.? [v2:172.21.15.136:6834/3349527335,v1:172.21.15.136:6835/3349527335] up:active 2023-07-20T01:27:48.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:47 smithi183 ceph-mon[98477]: from='client.? 172.21.15.60:0/459039830' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-20T01:27:48.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:47 smithi183 ceph-mon[98477]: daemon mds.f assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2023-07-20T01:27:48.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:47 smithi183 ceph-mon[98477]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2023-07-20T01:27:48.197 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:47 smithi183 ceph-mon[98477]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=f=up:creating} 2 up:standby-replay 6 up:standby 2023-07-20T01:27:48.198 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:47 smithi183 ceph-mon[98477]: daemon mds.f is now active in filesystem cephfs as rank 3 2023-07-20T01:27:48.269 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.289+0000 7f90e2153700 1 -- 172.21.15.60:0/1869042510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90dc102380 msgr2=0x7f90dc108c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:48.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.289+0000 7f90e2153700 1 --2- 172.21.15.60:0/1869042510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90dc102380 0x7f90dc108c30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f90d400b0d0 tx=0x7f90d400b3e0 comp rx=0 tx=0).stop 2023-07-20T01:27:48.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.289+0000 7f90e2153700 1 -- 172.21.15.60:0/1869042510 shutdown_connections 2023-07-20T01:27:48.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.289+0000 7f90e2153700 1 --2- 172.21.15.60:0/1869042510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90dc102380 0x7f90dc108c30 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:48.270 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.289+0000 7f90e2153700 1 --2- 172.21.15.60:0/1869042510 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90dc1007f0 0x7f90dc100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:48.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.289+0000 7f90e2153700 1 --2- 172.21.15.60:0/1869042510 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90dc1019e0 0x7f90dc101e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:48.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.289+0000 7f90e2153700 1 -- 172.21.15.60:0/1869042510 >> 172.21.15.60:0/1869042510 conn(0x7f90dc0fbd80 msgr2=0x7f90dc0fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:48.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.289+0000 7f90e2153700 1 -- 172.21.15.60:0/1869042510 shutdown_connections 2023-07-20T01:27:48.271 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.289+0000 7f90e2153700 1 -- 172.21.15.60:0/1869042510 wait complete. 2023-07-20T01:27:48.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.290+0000 7f90e2153700 1 Processor -- start 2023-07-20T01:27:48.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.290+0000 7f90e2153700 1 -- start start 2023-07-20T01:27:48.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.290+0000 7f90e2153700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90dc1007f0 0x7f90dc1964e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:48.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.290+0000 7f90e2153700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90dc1019e0 0x7f90dc196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:48.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.290+0000 7f90e2153700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90dc102380 0x7f90dc19da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:48.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.290+0000 7f90e2153700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc19e060 con 0x7f90dc1019e0 2023-07-20T01:27:48.272 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.290+0000 7f90e2153700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc19e1a0 con 0x7f90dc102380 2023-07-20T01:27:48.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.290+0000 7f90e2153700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc19e330 con 0x7f90dc1007f0 2023-07-20T01:27:48.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.290+0000 7f90daffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90dc1019e0 0x7f90dc196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:48.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.290+0000 7f90daffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90dc1019e0 0x7f90dc196a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43246/0 (socket says 172.21.15.60:43246) 2023-07-20T01:27:48.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.290+0000 7f90daffd700 1 -- 172.21.15.60:0/3501866757 learned_addr learned my addr 172.21.15.60:0/3501866757 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:27:48.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.290+0000 7f90db7fe700 1 --2- 172.21.15.60:0/3501866757 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90dc1007f0 0x7f90dc1964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:48.273 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.290+0000 7f90daffd700 1 -- 172.21.15.60:0/3501866757 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90dc1007f0 msgr2=0x7f90dc1964e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:48.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.290+0000 7f90daffd700 1 --2- 172.21.15.60:0/3501866757 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90dc1007f0 0x7f90dc1964e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:48.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.290+0000 7f90daffd700 1 -- 172.21.15.60:0/3501866757 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90dc102380 msgr2=0x7f90dc19da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:48.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.290+0000 7f90daffd700 1 --2- 172.21.15.60:0/3501866757 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90dc102380 0x7f90dc19da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:48.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.290+0000 7f90daffd700 1 -- 172.21.15.60:0/3501866757 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d4009d00 con 0x7f90dc1019e0 2023-07-20T01:27:48.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.291+0000 7f90daffd700 1 --2- 172.21.15.60:0/3501866757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90dc1019e0 0x7f90dc196a20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f90cc00d950 tx=0x7f90cc00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:48.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.291+0000 7f90d8ff9700 1 -- 172.21.15.60:0/3501866757 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90cc009b40 con 0x7f90dc1019e0 2023-07-20T01:27:48.274 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.291+0000 7f90d8ff9700 1 -- 172.21.15.60:0/3501866757 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f90cc009ca0 con 0x7f90dc1019e0 2023-07-20T01:27:48.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.291+0000 7f90e2153700 1 -- 172.21.15.60:0/3501866757 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90dc19e610 con 0x7f90dc1019e0 2023-07-20T01:27:48.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.291+0000 7f90d8ff9700 1 -- 172.21.15.60:0/3501866757 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90cc00f620 con 0x7f90dc1019e0 2023-07-20T01:27:48.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.291+0000 7f90e2153700 1 -- 172.21.15.60:0/3501866757 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90dc19eb20 con 0x7f90dc1019e0 2023-07-20T01:27:48.275 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.292+0000 7f90e2153700 1 -- 172.21.15.60:0/3501866757 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90dc107dd0 con 0x7f90dc1019e0 2023-07-20T01:27:48.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.293+0000 7f90d8ff9700 1 -- 172.21.15.60:0/3501866757 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f90cc010460 con 0x7f90dc1019e0 2023-07-20T01:27:48.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.294+0000 7f90d8ff9700 1 --2- 172.21.15.60:0/3501866757 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f90c40a1150 0x7f90c40a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:48.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.294+0000 7f90db7fe700 1 --2- 172.21.15.60:0/3501866757 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f90c40a1150 0x7f90c40a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:48.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.294+0000 7f90d8ff9700 1 -- 172.21.15.60:0/3501866757 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f90cc0ccb70 con 0x7f90dc1019e0 2023-07-20T01:27:48.276 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.294+0000 7f90db7fe700 1 --2- 172.21.15.60:0/3501866757 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f90c40a1150 0x7f90c40a3610 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f90d0005950 tx=0x7f90d00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:48.277 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.298+0000 7f90d8ff9700 1 -- 172.21.15.60:0/3501866757 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f90cc096410 con 0x7f90dc1019e0 2023-07-20T01:27:48.394 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:47 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/459039830' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-20T01:27:48.394 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:47 smithi136 ceph-mon[98083]: mds.? [v2:172.21.15.136:6834/3349527335,v1:172.21.15.136:6835/3349527335] up:active 2023-07-20T01:27:48.394 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:47 smithi136 ceph-mon[98083]: from='client.? 172.21.15.60:0/459039830' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-20T01:27:48.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:47 smithi136 ceph-mon[98083]: daemon mds.f assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2023-07-20T01:27:48.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:47 smithi136 ceph-mon[98083]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2023-07-20T01:27:48.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:47 smithi136 ceph-mon[98083]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=f=up:creating} 2 up:standby-replay 6 up:standby 2023-07-20T01:27:48.395 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:47 smithi136 ceph-mon[98083]: daemon mds.f is now active in filesystem cephfs as rank 3 2023-07-20T01:27:48.424 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:27:48 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:27:48.122+0000 7f2cd24a8700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T01:27:48.424 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:27:48 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:27:48.122+0000 7f2cd24a8700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T01:27:48.424 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:27:48 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:27:48.122+0000 7f2cd24a8700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T01:27:48.424 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:27:48 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:27:48.122+0000 7f2cd24a8700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T01:27:48.424 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:27:48 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:27:48.122+0000 7f2cd24a8700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-20T01:27:48.510 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.532+0000 7f90e2153700 1 -- 172.21.15.60:0/3501866757 --> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "group_name": "", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f90dc060c40 con 0x7f90c40a1150 2023-07-20T01:27:48.538 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.561+0000 7f90d8ff9700 1 -- 172.21.15.60:0/3501866757 <== mgr.14152 v2:172.21.15.60:6800/1961833180 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f90dc060c40 con 0x7f90c40a1150 2023-07-20T01:27:48.540 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.563+0000 7f90e2153700 1 -- 172.21.15.60:0/3501866757 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f90c40a1150 msgr2=0x7f90c40a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:48.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.563+0000 7f90e2153700 1 --2- 172.21.15.60:0/3501866757 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f90c40a1150 0x7f90c40a3610 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f90d0005950 tx=0x7f90d00058e0 comp rx=0 tx=0).stop 2023-07-20T01:27:48.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.563+0000 7f90e2153700 1 -- 172.21.15.60:0/3501866757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90dc1019e0 msgr2=0x7f90dc196a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:48.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.563+0000 7f90e2153700 1 --2- 172.21.15.60:0/3501866757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90dc1019e0 0x7f90dc196a20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f90cc00d950 tx=0x7f90cc00dd10 comp rx=0 tx=0).stop 2023-07-20T01:27:48.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.563+0000 7f90e2153700 1 -- 172.21.15.60:0/3501866757 shutdown_connections 2023-07-20T01:27:48.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.563+0000 7f90e2153700 1 --2- 172.21.15.60:0/3501866757 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7f90c40a1150 0x7f90c40a3610 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:48.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.563+0000 7f90e2153700 1 --2- 172.21.15.60:0/3501866757 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90dc102380 0x7f90dc19da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:48.541 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.563+0000 7f90e2153700 1 --2- 172.21.15.60:0/3501866757 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90dc1007f0 0x7f90dc1964e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:48.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.563+0000 7f90e2153700 1 --2- 172.21.15.60:0/3501866757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90dc1019e0 0x7f90dc196a20 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:48.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.563+0000 7f90e2153700 1 -- 172.21.15.60:0/3501866757 >> 172.21.15.60:0/3501866757 conn(0x7f90dc0fbd80 msgr2=0x7f90dc104f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:48.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.563+0000 7f90e2153700 1 -- 172.21.15.60:0/3501866757 shutdown_connections 2023-07-20T01:27:48.542 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.563+0000 7f90e2153700 1 -- 172.21.15.60:0/3501866757 wait complete. 2023-07-20T01:27:48.553 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name=qa sv_0 2023-07-20T01:27:48.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.707+0000 7ff80e35a700 1 -- 172.21.15.60:0/168339263 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff808103b10 msgr2=0x7ff808105f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:48.685 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.707+0000 7ff80e35a700 1 --2- 172.21.15.60:0/168339263 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff808103b10 0x7ff808105f00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc009990 tx=0x7ff7fc009ca0 comp rx=0 tx=0).stop 2023-07-20T01:27:48.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.707+0000 7ff80e35a700 1 -- 172.21.15.60:0/168339263 shutdown_connections 2023-07-20T01:27:48.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.707+0000 7ff80e35a700 1 --2- 172.21.15.60:0/168339263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff808106440 0x7ff8081088b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:48.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.707+0000 7ff80e35a700 1 --2- 172.21.15.60:0/168339263 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8081011e0 0x7ff8081035d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:48.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.707+0000 7ff80e35a700 1 --2- 172.21.15.60:0/168339263 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff808103b10 0x7ff808105f00 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:48.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.707+0000 7ff80e35a700 1 -- 172.21.15.60:0/168339263 >> 172.21.15.60:0/168339263 conn(0x7ff8080fabd0 msgr2=0x7ff8080fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-20T01:27:48.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.707+0000 7ff80e35a700 1 -- 172.21.15.60:0/168339263 shutdown_connections 2023-07-20T01:27:48.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.707+0000 7ff80e35a700 1 -- 172.21.15.60:0/168339263 wait complete. 2023-07-20T01:27:48.686 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.708+0000 7ff80e35a700 1 Processor -- start 2023-07-20T01:27:48.687 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.708+0000 7ff80e35a700 1 -- start start 2023-07-20T01:27:48.689 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.708+0000 7ff80e35a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8081011e0 0x7ff808196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:48.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.708+0000 7ff80e35a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff808103b10 0x7ff8081968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:48.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.708+0000 7ff80e35a700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff808106440 0x7ff80819d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:48.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.708+0000 7ff80e35a700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80819df10 con 0x7ff8081011e0 2023-07-20T01:27:48.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.708+0000 7ff80e35a700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80819e050 con 0x7ff808103b10 2023-07-20T01:27:48.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.708+0000 7ff80e35a700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80819e1e0 con 0x7ff808106440 2023-07-20T01:27:48.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.708+0000 7ff807fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8081011e0 0x7ff808196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:48.690 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.709+0000 7ff807fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8081011e0 0x7ff808196390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43260/0 (socket says 172.21.15.60:43260) 2023-07-20T01:27:48.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.709+0000 7ff807fff700 1 -- 172.21.15.60:0/3223665486 learned_addr learned my addr 172.21.15.60:0/3223665486 (peer_addr_for_me v2:172.21.15.60:0/0) 2023-07-20T01:27:48.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.709+0000 7ff8077fe700 1 --2- 172.21.15.60:0/3223665486 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff808103b10 0x7ff8081968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:48.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.709+0000 7ff80c8f7700 1 --2- 172.21.15.60:0/3223665486 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff808106440 0x7ff80819d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:48.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.709+0000 7ff807fff700 1 -- 172.21.15.60:0/3223665486 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff808106440 msgr2=0x7ff80819d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:48.691 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.709+0000 7ff807fff700 1 --2- 172.21.15.60:0/3223665486 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff808106440 0x7ff80819d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:48.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.709+0000 7ff807fff700 1 -- 172.21.15.60:0/3223665486 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff808103b10 msgr2=0x7ff8081968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-20T01:27:48.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.709+0000 7ff807fff700 1 --2- 172.21.15.60:0/3223665486 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff808103b10 0x7ff8081968d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-20T01:27:48.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.709+0000 7ff807fff700 1 -- 172.21.15.60:0/3223665486 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7fc0095f0 con 0x7ff8081011e0 2023-07-20T01:27:48.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.709+0000 7ff807fff700 1 --2- 172.21.15.60:0/3223665486 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8081011e0 0x7ff808196390 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff7f800d950 tx=0x7ff7f800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:48.692 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.709+0000 7ff8057fa700 1 -- 172.21.15.60:0/3223665486 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f8009940 con 0x7ff8081011e0 2023-07-20T01:27:48.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.709+0000 7ff8057fa700 1 -- 172.21.15.60:0/3223665486 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff7f8010460 con 0x7ff8081011e0 2023-07-20T01:27:48.693 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.709+0000 7ff80e35a700 1 -- 172.21.15.60:0/3223665486 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff80819e4c0 con 0x7ff8081011e0 2023-07-20T01:27:48.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.709+0000 7ff8057fa700 1 -- 172.21.15.60:0/3223665486 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f800f620 con 0x7ff8081011e0 2023-07-20T01:27:48.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.709+0000 7ff80e35a700 1 -- 172.21.15.60:0/3223665486 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff80819e9d0 con 0x7ff8081011e0 2023-07-20T01:27:48.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.710+0000 7ff80e35a700 1 -- 172.21.15.60:0/3223665486 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8080fc750 con 0x7ff8081011e0 2023-07-20T01:27:48.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.711+0000 7ff8057fa700 1 -- 172.21.15.60:0/3223665486 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff7f80105d0 con 0x7ff8081011e0 2023-07-20T01:27:48.694 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.712+0000 7ff8057fa700 1 --2- 172.21.15.60:0/3223665486 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7ff7f00a1150 0x7ff7f00a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-20T01:27:48.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.712+0000 7ff8077fe700 1 --2- 172.21.15.60:0/3223665486 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7ff7f00a1150 0x7ff7f00a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-20T01:27:48.695 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.712+0000 7ff8057fa700 1 -- 172.21.15.60:0/3223665486 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ff7f80cc610 con 0x7ff8081011e0 2023-07-20T01:27:48.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.712+0000 7ff8077fe700 1 --2- 172.21.15.60:0/3223665486 >> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] conn(0x7ff7f00a1150 0x7ff7f00a3610 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc00b5c0 tx=0x7ff7fc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-20T01:27:48.696 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.716+0000 7ff8057fa700 1 -- 172.21.15.60:0/3223665486 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff7f8095eb0 con 0x7ff8081011e0 2023-07-20T01:27:48.934 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.956+0000 7ff80e35a700 1 -- 172.21.15.60:0/3223665486 --> [v2:172.21.15.60:6800/1961833180,v1:172.21.15.60:6801/1961833180] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7ff808060c40 con 0x7ff7f00a1150 2023-07-20T01:27:48.936 INFO:teuthology.orchestra.run.smithi060.stderr:2023-07-20T01:27:48.959+0000 7ff8057fa700 1 -- 172.21.15.60:0/3223665486 <== mgr.14152 v2:172.21.15.60:6800/1961833180 1 ==== mgr_command_reply(tid 0: -2 subvolume 'sv_0' does not exist) v1 ==== 39+0+0 (secure 0 0 0) 0x7ff808060c40 con 0x7ff7f00a1150 2023-07-20T01:27:48.938 INFO:teuthology.orchestra.run.smithi060.stderr:Error ENOENT: subvolume 'sv_0' does not exist 2023-07-20T01:27:48.941 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-07-20T01:27:48.942 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephadm.py", line 980, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/filesystem.py", line 535, in __init__ self.create() File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/filesystem.py", line 773, in create subvol_path = self.get_ceph_cmd_stdout( File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/cephfs_test_case.py", line 73, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/cephfs_test_case.py", line 58, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 2: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name=qa sv_0' 2023-07-20T01:27:48.943 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T01:27:48.970 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T01:27:49.008 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T01:27:49.046 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-07-20T01:27:49.046 DEBUG:teuthology.orchestra.run.smithi060:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-20T01:27:49.064 DEBUG:teuthology.orchestra.run.smithi136:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-20T01:27:49.083 DEBUG:teuthology.orchestra.run.smithi183:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-20T01:27:49.105 INFO:tasks.cephadm:Stopping all daemons... 2023-07-20T01:27:49.105 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-07-20T01:27:49.106 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mon.a 2023-07-20T01:27:49.424 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:49 smithi060 systemd[1]: Stopping Ceph mon.a for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:27:49.425 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:27:48 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:27:48.959+0000 7f2cd24a8700 -1 mgr.server reply reply (2) No such file or directory subvolume 'sv_0' does not exist 2023-07-20T01:27:49.806 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:49 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mon-a[96499]: 2023-07-20T01:27:49.489+0000 7f9cab3b4700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:27:49.807 INFO:journalctl@ceph.mon.a.smithi060.stdout:Jul 20 01:27:49 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mon-a[96499]: 2023-07-20T01:27:49.489+0000 7f9cab3b4700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2023-07-20T01:27:50.011 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mon.a.service' 2023-07-20T01:27:50.074 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:27:50.075 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-07-20T01:27:50.076 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2023-07-20T01:27:50.078 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mon.b 2023-07-20T01:27:50.394 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:50 smithi136 systemd[1]: Stopping Ceph mon.b for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:27:50.713 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:50 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mon-b[98058]: 2023-07-20T01:27:50.443+0000 7f5eddece700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:27:50.713 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:50 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mon-b[98058]: 2023-07-20T01:27:50.443+0000 7f5eddece700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2023-07-20T01:27:51.081 INFO:journalctl@ceph.mon.b.smithi136.stdout:Jul 20 01:27:50 smithi136 bash[124379]: ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mon-b 2023-07-20T01:27:51.089 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mon.b.service' 2023-07-20T01:27:51.151 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:27:51.152 INFO:tasks.cephadm.mon.c:Stopped mon.b 2023-07-20T01:27:51.152 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2023-07-20T01:27:51.153 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mon.c 2023-07-20T01:27:51.447 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:51 smithi183 systemd[1]: Stopping Ceph mon.c for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:27:51.847 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:51 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mon-c[98450]: 2023-07-20T01:27:51.538+0000 7ff79d2fc700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:27:51.847 INFO:journalctl@ceph.mon.c.smithi183.stdout:Jul 20 01:27:51 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mon-c[98450]: 2023-07-20T01:27:51.538+0000 7ff79d2fc700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2023-07-20T01:27:52.056 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mon.c.service' 2023-07-20T01:27:52.122 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:27:52.123 INFO:tasks.cephadm.mon.c:Stopped mon.c 2023-07-20T01:27:52.123 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-07-20T01:27:52.124 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mgr.x 2023-07-20T01:27:52.424 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:27:52 smithi060 systemd[1]: Stopping Ceph mgr.x for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:27:52.764 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:27:52 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x[96770]: 2023-07-20T01:27:52.513+0000 7f2d640dc700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:27:53.029 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Jul 20 01:27:52 smithi060 bash[133333]: ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-x 2023-07-20T01:27:53.036 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mgr.x.service' 2023-07-20T01:27:53.078 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:27:53.079 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-07-20T01:27:53.079 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2023-07-20T01:27:53.079 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mgr.y 2023-07-20T01:27:53.394 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:27:53 smithi136 systemd[1]: Stopping Ceph mgr.y for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:27:53.779 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:27:53 smithi136 bash[124761]: ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-y 2023-07-20T01:27:53.960 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Jul 20 01:27:53 smithi136 systemd[1]: ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mgr.y.service: Main process exited, code=exited, status=143/n/a 2023-07-20T01:27:53.963 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mgr.y.service' 2023-07-20T01:27:54.001 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:27:54.002 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2023-07-20T01:27:54.002 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2023-07-20T01:27:54.003 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mgr.z 2023-07-20T01:27:54.429 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:27:54 smithi183 systemd[1]: Stopping Ceph mgr.z for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:27:54.697 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:27:54 smithi183 bash[126091]: ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mgr-z 2023-07-20T01:27:54.896 INFO:journalctl@ceph.mgr.z.smithi183.stdout:Jul 20 01:27:54 smithi183 systemd[1]: ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mgr.z.service: Main process exited, code=exited, status=143/n/a 2023-07-20T01:27:54.901 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mgr.z.service' 2023-07-20T01:27:54.939 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:27:54.940 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2023-07-20T01:27:54.940 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-07-20T01:27:54.941 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.0 2023-07-20T01:27:55.308 INFO:journalctl@ceph.osd.0.smithi060.stdout:Jul 20 01:27:55 smithi060 systemd[1]: Stopping Ceph osd.0 for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:27:55.674 INFO:journalctl@ceph.osd.0.smithi060.stdout:Jul 20 01:27:55 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-0[109432]: 2023-07-20T01:27:55.330+0000 7f4ea11e0700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:27:55.674 INFO:journalctl@ceph.osd.0.smithi060.stdout:Jul 20 01:27:55 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-0[109432]: 2023-07-20T01:27:55.330+0000 7f4ea11e0700 -1 osd.0 82 *** Got signal Terminated *** 2023-07-20T01:27:55.674 INFO:journalctl@ceph.osd.0.smithi060.stdout:Jul 20 01:27:55 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-0[109432]: 2023-07-20T01:27:55.330+0000 7f4ea11e0700 -1 osd.0 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T01:28:01.082 INFO:journalctl@ceph.osd.0.smithi060.stdout:Jul 20 01:28:01 smithi060 bash[133474]: ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-0 2023-07-20T01:28:02.564 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.0.service' 2023-07-20T01:28:02.609 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:28:02.610 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-07-20T01:28:02.610 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-07-20T01:28:02.611 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.1 2023-07-20T01:28:02.894 INFO:journalctl@ceph.osd.1.smithi136.stdout:Jul 20 01:28:02 smithi136 systemd[1]: Stopping Ceph osd.1 for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:28:03.394 INFO:journalctl@ceph.osd.1.smithi136.stdout:Jul 20 01:28:03 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-1[103151]: 2023-07-20T01:28:03.018+0000 7f21495ac700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:28:03.394 INFO:journalctl@ceph.osd.1.smithi136.stdout:Jul 20 01:28:03 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-1[103151]: 2023-07-20T01:28:03.018+0000 7f21495ac700 -1 osd.1 82 *** Got signal Terminated *** 2023-07-20T01:28:03.394 INFO:journalctl@ceph.osd.1.smithi136.stdout:Jul 20 01:28:03 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-1[103151]: 2023-07-20T01:28:03.018+0000 7f21495ac700 -1 osd.1 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T01:28:08.856 INFO:journalctl@ceph.osd.1.smithi136.stdout:Jul 20 01:28:08 smithi136 bash[124904]: ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-1 2023-07-20T01:28:10.118 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.1.service' 2023-07-20T01:28:10.160 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:28:10.161 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-07-20T01:28:10.162 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-07-20T01:28:10.163 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.2 2023-07-20T01:28:10.446 INFO:journalctl@ceph.osd.2.smithi183.stdout:Jul 20 01:28:10 smithi183 systemd[1]: Stopping Ceph osd.2 for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:28:10.946 INFO:journalctl@ceph.osd.2.smithi183.stdout:Jul 20 01:28:10 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-2[104265]: 2023-07-20T01:28:10.543+0000 7fcc38d98700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:28:10.947 INFO:journalctl@ceph.osd.2.smithi183.stdout:Jul 20 01:28:10 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-2[104265]: 2023-07-20T01:28:10.543+0000 7fcc38d98700 -1 osd.2 82 *** Got signal Terminated *** 2023-07-20T01:28:10.947 INFO:journalctl@ceph.osd.2.smithi183.stdout:Jul 20 01:28:10 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-2[104265]: 2023-07-20T01:28:10.543+0000 7fcc38d98700 -1 osd.2 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T01:28:16.446 INFO:journalctl@ceph.osd.2.smithi183.stdout:Jul 20 01:28:16 smithi183 bash[126237]: ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-2 2023-07-20T01:28:17.797 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.2.service' 2023-07-20T01:28:17.838 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:28:17.838 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-07-20T01:28:17.839 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-07-20T01:28:17.839 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.3 2023-07-20T01:28:18.173 INFO:journalctl@ceph.osd.3.smithi060.stdout:Jul 20 01:28:17 smithi060 systemd[1]: Stopping Ceph osd.3 for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:28:18.673 INFO:journalctl@ceph.osd.3.smithi060.stdout:Jul 20 01:28:18 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-3[114879]: 2023-07-20T01:28:18.227+0000 7f7174e94700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:28:18.674 INFO:journalctl@ceph.osd.3.smithi060.stdout:Jul 20 01:28:18 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-3[114879]: 2023-07-20T01:28:18.227+0000 7f7174e94700 -1 osd.3 82 *** Got signal Terminated *** 2023-07-20T01:28:18.674 INFO:journalctl@ceph.osd.3.smithi060.stdout:Jul 20 01:28:18 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-3[114879]: 2023-07-20T01:28:18.227+0000 7f7174e94700 -1 osd.3 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T01:28:24.173 INFO:journalctl@ceph.osd.3.smithi060.stdout:Jul 20 01:28:23 smithi060 bash[133746]: ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-3 2023-07-20T01:28:25.362 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.3.service' 2023-07-20T01:28:25.403 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:28:25.404 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-07-20T01:28:25.404 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-07-20T01:28:25.405 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.4 2023-07-20T01:28:25.783 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jul 20 01:28:25 smithi136 systemd[1]: Stopping Ceph osd.4 for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:28:26.143 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jul 20 01:28:25 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-4[108572]: 2023-07-20T01:28:25.805+0000 7f881f1a1700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:28:26.144 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jul 20 01:28:25 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-4[108572]: 2023-07-20T01:28:25.805+0000 7f881f1a1700 -1 osd.4 82 *** Got signal Terminated *** 2023-07-20T01:28:26.144 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jul 20 01:28:25 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-4[108572]: 2023-07-20T01:28:25.805+0000 7f881f1a1700 -1 osd.4 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T01:28:31.643 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jul 20 01:28:31 smithi136 bash[125172]: ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-4 2023-07-20T01:28:32.805 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.4.service' 2023-07-20T01:28:32.845 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:28:32.846 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-07-20T01:28:32.846 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-07-20T01:28:32.846 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.5 2023-07-20T01:28:33.196 INFO:journalctl@ceph.osd.5.smithi183.stdout:Jul 20 01:28:32 smithi183 systemd[1]: Stopping Ceph osd.5 for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:28:33.696 INFO:journalctl@ceph.osd.5.smithi183.stdout:Jul 20 01:28:33 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-5[109748]: 2023-07-20T01:28:33.230+0000 7f5deac3f700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:28:33.696 INFO:journalctl@ceph.osd.5.smithi183.stdout:Jul 20 01:28:33 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-5[109748]: 2023-07-20T01:28:33.230+0000 7f5deac3f700 -1 osd.5 82 *** Got signal Terminated *** 2023-07-20T01:28:33.697 INFO:journalctl@ceph.osd.5.smithi183.stdout:Jul 20 01:28:33 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-5[109748]: 2023-07-20T01:28:33.230+0000 7f5deac3f700 -1 osd.5 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T01:28:38.884 INFO:journalctl@ceph.osd.5.smithi183.stdout:Jul 20 01:28:38 smithi183 bash[126518]: ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-5 2023-07-20T01:28:40.157 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.5.service' 2023-07-20T01:28:40.198 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:28:40.198 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-07-20T01:28:40.199 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2023-07-20T01:28:40.200 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.6 2023-07-20T01:28:40.559 INFO:journalctl@ceph.osd.6.smithi060.stdout:Jul 20 01:28:40 smithi060 systemd[1]: Stopping Ceph osd.6 for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:28:40.923 INFO:journalctl@ceph.osd.6.smithi060.stdout:Jul 20 01:28:40 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-6[120307]: 2023-07-20T01:28:40.583+0000 7fc527aa1700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:28:40.923 INFO:journalctl@ceph.osd.6.smithi060.stdout:Jul 20 01:28:40 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-6[120307]: 2023-07-20T01:28:40.583+0000 7fc527aa1700 -1 osd.6 82 *** Got signal Terminated *** 2023-07-20T01:28:40.923 INFO:journalctl@ceph.osd.6.smithi060.stdout:Jul 20 01:28:40 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-6[120307]: 2023-07-20T01:28:40.583+0000 7fc527aa1700 -1 osd.6 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T01:28:46.345 INFO:journalctl@ceph.osd.6.smithi060.stdout:Jul 20 01:28:46 smithi060 bash[134017]: ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-6 2023-07-20T01:28:47.618 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.6.service' 2023-07-20T01:28:47.661 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:28:47.661 INFO:tasks.cephadm.osd.6:Stopped osd.6 2023-07-20T01:28:47.662 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2023-07-20T01:28:47.662 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.7 2023-07-20T01:28:48.018 INFO:journalctl@ceph.osd.7.smithi136.stdout:Jul 20 01:28:47 smithi136 systemd[1]: Stopping Ceph osd.7 for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:28:48.393 INFO:journalctl@ceph.osd.7.smithi136.stdout:Jul 20 01:28:48 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-7[113977]: 2023-07-20T01:28:48.041+0000 7f4abfef7700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:28:48.394 INFO:journalctl@ceph.osd.7.smithi136.stdout:Jul 20 01:28:48 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-7[113977]: 2023-07-20T01:28:48.041+0000 7f4abfef7700 -1 osd.7 82 *** Got signal Terminated *** 2023-07-20T01:28:48.394 INFO:journalctl@ceph.osd.7.smithi136.stdout:Jul 20 01:28:48 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-7[113977]: 2023-07-20T01:28:48.041+0000 7f4abfef7700 -1 osd.7 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T01:28:53.699 INFO:journalctl@ceph.osd.7.smithi136.stdout:Jul 20 01:28:53 smithi136 bash[125445]: ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-7 2023-07-20T01:28:54.965 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.7.service' 2023-07-20T01:28:55.004 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:28:55.005 INFO:tasks.cephadm.osd.7:Stopped osd.7 2023-07-20T01:28:55.005 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2023-07-20T01:28:55.005 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.8 2023-07-20T01:28:55.362 INFO:journalctl@ceph.osd.8.smithi183.stdout:Jul 20 01:28:55 smithi183 systemd[1]: Stopping Ceph osd.8 for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:28:55.696 INFO:journalctl@ceph.osd.8.smithi183.stdout:Jul 20 01:28:55 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-8[115251]: 2023-07-20T01:28:55.385+0000 7fd547090700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:28:55.696 INFO:journalctl@ceph.osd.8.smithi183.stdout:Jul 20 01:28:55 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-8[115251]: 2023-07-20T01:28:55.385+0000 7fd547090700 -1 osd.8 82 *** Got signal Terminated *** 2023-07-20T01:28:55.696 INFO:journalctl@ceph.osd.8.smithi183.stdout:Jul 20 01:28:55 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-8[115251]: 2023-07-20T01:28:55.385+0000 7fd547090700 -1 osd.8 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T01:29:01.196 INFO:journalctl@ceph.osd.8.smithi183.stdout:Jul 20 01:29:00 smithi183 bash[126780]: ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-8 2023-07-20T01:29:02.143 INFO:journalctl@ceph.osd.10.smithi136.stdout:Jul 20 01:29:01 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-10[119505]: 2023-07-20T01:29:01.679+0000 7fbf3ed38700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:57.653079+0000 front 2023-07-20T01:27:57.653056+0000 (oldest deadline 2023-07-20T01:29:01.153098+0000) 2023-07-20T01:29:02.245 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.8.service' 2023-07-20T01:29:02.286 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:29:02.286 INFO:tasks.cephadm.osd.8:Stopped osd.8 2023-07-20T01:29:02.287 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2023-07-20T01:29:02.287 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.9 2023-07-20T01:29:02.673 INFO:journalctl@ceph.osd.9.smithi060.stdout:Jul 20 01:29:02 smithi060 systemd[1]: Stopping Ceph osd.9 for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:29:03.143 INFO:journalctl@ceph.osd.10.smithi136.stdout:Jul 20 01:29:02 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-10[119505]: 2023-07-20T01:29:02.719+0000 7fbf3ed38700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:57.653079+0000 front 2023-07-20T01:27:57.653056+0000 (oldest deadline 2023-07-20T01:29:01.153098+0000) 2023-07-20T01:29:03.173 INFO:journalctl@ceph.osd.9.smithi060.stdout:Jul 20 01:29:02 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-9[125860]: 2023-07-20T01:29:02.740+0000 7f1831745700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:29:03.173 INFO:journalctl@ceph.osd.9.smithi060.stdout:Jul 20 01:29:02 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-9[125860]: 2023-07-20T01:29:02.740+0000 7f1831745700 -1 osd.9 82 *** Got signal Terminated *** 2023-07-20T01:29:03.173 INFO:journalctl@ceph.osd.9.smithi060.stdout:Jul 20 01:29:02 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-9[125860]: 2023-07-20T01:29:02.740+0000 7f1831745700 -1 osd.9 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T01:29:04.143 INFO:journalctl@ceph.osd.10.smithi136.stdout:Jul 20 01:29:03 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-10[119505]: 2023-07-20T01:29:03.751+0000 7fbf3ed38700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:57.653079+0000 front 2023-07-20T01:27:57.653056+0000 (oldest deadline 2023-07-20T01:29:01.153098+0000) 2023-07-20T01:29:04.923 INFO:journalctl@ceph.osd.9.smithi060.stdout:Jul 20 01:29:04 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-9[125860]: 2023-07-20T01:29:04.605+0000 7f182dd3c700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:58.578132+0000 front 2023-07-20T01:27:58.578064+0000 (oldest deadline 2023-07-20T01:29:04.478233+0000) 2023-07-20T01:29:05.143 INFO:journalctl@ceph.osd.10.smithi136.stdout:Jul 20 01:29:04 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-10[119505]: 2023-07-20T01:29:04.738+0000 7fbf3ed38700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:57.653079+0000 front 2023-07-20T01:27:57.653056+0000 (oldest deadline 2023-07-20T01:29:01.153098+0000) 2023-07-20T01:29:05.196 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:04 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:04.898+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:59.308083+0000 front 2023-07-20T01:27:59.308175+0000 (oldest deadline 2023-07-20T01:29:04.008156+0000) 2023-07-20T01:29:05.923 INFO:journalctl@ceph.osd.9.smithi060.stdout:Jul 20 01:29:05 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-9[125860]: 2023-07-20T01:29:05.633+0000 7f182dd3c700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:58.578132+0000 front 2023-07-20T01:27:58.578064+0000 (oldest deadline 2023-07-20T01:29:04.478233+0000) 2023-07-20T01:29:06.143 INFO:journalctl@ceph.osd.10.smithi136.stdout:Jul 20 01:29:05 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-10[119505]: 2023-07-20T01:29:05.703+0000 7fbf3ed38700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:57.653079+0000 front 2023-07-20T01:27:57.653056+0000 (oldest deadline 2023-07-20T01:29:01.153098+0000) 2023-07-20T01:29:06.196 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:05 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:05.888+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:59.308083+0000 front 2023-07-20T01:27:59.308175+0000 (oldest deadline 2023-07-20T01:29:04.008156+0000) 2023-07-20T01:29:06.893 INFO:journalctl@ceph.osd.10.smithi136.stdout:Jul 20 01:29:06 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-10[119505]: 2023-07-20T01:29:06.664+0000 7fbf3ed38700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:57.653079+0000 front 2023-07-20T01:27:57.653056+0000 (oldest deadline 2023-07-20T01:29:01.153098+0000) 2023-07-20T01:29:06.922 INFO:journalctl@ceph.osd.9.smithi060.stdout:Jul 20 01:29:06 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-9[125860]: 2023-07-20T01:29:06.612+0000 7f182dd3c700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:58.578132+0000 front 2023-07-20T01:27:58.578064+0000 (oldest deadline 2023-07-20T01:29:04.478233+0000) 2023-07-20T01:29:07.196 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:06 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:06.843+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:59.308083+0000 front 2023-07-20T01:27:59.308175+0000 (oldest deadline 2023-07-20T01:29:04.008156+0000) 2023-07-20T01:29:07.893 INFO:journalctl@ceph.osd.10.smithi136.stdout:Jul 20 01:29:07 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-10[119505]: 2023-07-20T01:29:07.628+0000 7fbf3ed38700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:57.653079+0000 front 2023-07-20T01:27:57.653056+0000 (oldest deadline 2023-07-20T01:29:01.153098+0000) 2023-07-20T01:29:07.923 INFO:journalctl@ceph.osd.9.smithi060.stdout:Jul 20 01:29:07 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-9[125860]: 2023-07-20T01:29:07.612+0000 7f182dd3c700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:58.578132+0000 front 2023-07-20T01:27:58.578064+0000 (oldest deadline 2023-07-20T01:29:04.478233+0000) 2023-07-20T01:29:08.196 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:07 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:07.888+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:59.308083+0000 front 2023-07-20T01:27:59.308175+0000 (oldest deadline 2023-07-20T01:29:04.008156+0000) 2023-07-20T01:29:08.423 INFO:journalctl@ceph.osd.9.smithi060.stdout:Jul 20 01:29:08 smithi060 bash[134279]: ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-9 2023-07-20T01:29:08.893 INFO:journalctl@ceph.osd.10.smithi136.stdout:Jul 20 01:29:08 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-10[119505]: 2023-07-20T01:29:08.586+0000 7fbf3ed38700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:57.653079+0000 front 2023-07-20T01:27:57.653056+0000 (oldest deadline 2023-07-20T01:29:01.153098+0000) 2023-07-20T01:29:09.195 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:08 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:08.877+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:59.308083+0000 front 2023-07-20T01:27:59.308175+0000 (oldest deadline 2023-07-20T01:29:04.008156+0000) 2023-07-20T01:29:09.769 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.9.service' 2023-07-20T01:29:09.810 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:29:09.811 INFO:tasks.cephadm.osd.9:Stopped osd.9 2023-07-20T01:29:09.811 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2023-07-20T01:29:09.811 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.10 2023-07-20T01:29:09.841 INFO:journalctl@ceph.osd.10.smithi136.stdout:Jul 20 01:29:09 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-10[119505]: 2023-07-20T01:29:09.593+0000 7fbf3ed38700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:57.653079+0000 front 2023-07-20T01:27:57.653056+0000 (oldest deadline 2023-07-20T01:29:01.153098+0000) 2023-07-20T01:29:10.143 INFO:journalctl@ceph.osd.10.smithi136.stdout:Jul 20 01:29:09 smithi136 systemd[1]: Stopping Ceph osd.10 for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:29:10.196 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:09 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:09.898+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:59.308083+0000 front 2023-07-20T01:27:59.308175+0000 (oldest deadline 2023-07-20T01:29:04.008156+0000) 2023-07-20T01:29:10.589 INFO:journalctl@ceph.osd.10.smithi136.stdout:Jul 20 01:29:10 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-10[119505]: 2023-07-20T01:29:10.195+0000 7fbf44745700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:29:10.589 INFO:journalctl@ceph.osd.10.smithi136.stdout:Jul 20 01:29:10 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-10[119505]: 2023-07-20T01:29:10.195+0000 7fbf44745700 -1 osd.10 82 *** Got signal Terminated *** 2023-07-20T01:29:10.589 INFO:journalctl@ceph.osd.10.smithi136.stdout:Jul 20 01:29:10 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-10[119505]: 2023-07-20T01:29:10.195+0000 7fbf44745700 -1 osd.10 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T01:29:10.893 INFO:journalctl@ceph.osd.10.smithi136.stdout:Jul 20 01:29:10 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-10[119505]: 2023-07-20T01:29:10.613+0000 7fbf3ed38700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:57.653079+0000 front 2023-07-20T01:27:57.653056+0000 (oldest deadline 2023-07-20T01:29:01.153098+0000) 2023-07-20T01:29:10.893 INFO:journalctl@ceph.osd.10.smithi136.stdout:Jul 20 01:29:10 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-10[119505]: 2023-07-20T01:29:10.613+0000 7fbf3ed38700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2023-07-20T01:28:06.255361+0000 front 2023-07-20T01:28:06.255385+0000 (oldest deadline 2023-07-20T01:29:09.755472+0000) 2023-07-20T01:29:11.196 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:10 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:10.912+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:59.308083+0000 front 2023-07-20T01:27:59.308175+0000 (oldest deadline 2023-07-20T01:29:04.008156+0000) 2023-07-20T01:29:11.196 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:10 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:10.912+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2023-07-20T01:28:04.008814+0000 front 2023-07-20T01:28:04.008729+0000 (oldest deadline 2023-07-20T01:29:09.908820+0000) 2023-07-20T01:29:11.893 INFO:journalctl@ceph.osd.10.smithi136.stdout:Jul 20 01:29:11 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-10[119505]: 2023-07-20T01:29:11.663+0000 7fbf3ed38700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:57.653079+0000 front 2023-07-20T01:27:57.653056+0000 (oldest deadline 2023-07-20T01:29:01.153098+0000) 2023-07-20T01:29:11.893 INFO:journalctl@ceph.osd.10.smithi136.stdout:Jul 20 01:29:11 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-10[119505]: 2023-07-20T01:29:11.663+0000 7fbf3ed38700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2023-07-20T01:28:06.255361+0000 front 2023-07-20T01:28:06.255385+0000 (oldest deadline 2023-07-20T01:29:09.755472+0000) 2023-07-20T01:29:12.196 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:11 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:11.954+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:59.308083+0000 front 2023-07-20T01:27:59.308175+0000 (oldest deadline 2023-07-20T01:29:04.008156+0000) 2023-07-20T01:29:12.196 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:11 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:11.954+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2023-07-20T01:28:04.008814+0000 front 2023-07-20T01:28:04.008729+0000 (oldest deadline 2023-07-20T01:29:09.908820+0000) 2023-07-20T01:29:12.893 INFO:journalctl@ceph.osd.10.smithi136.stdout:Jul 20 01:29:12 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-10[119505]: 2023-07-20T01:29:12.650+0000 7fbf3ed38700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:57.653079+0000 front 2023-07-20T01:27:57.653056+0000 (oldest deadline 2023-07-20T01:29:01.153098+0000) 2023-07-20T01:29:12.893 INFO:journalctl@ceph.osd.10.smithi136.stdout:Jul 20 01:29:12 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-10[119505]: 2023-07-20T01:29:12.650+0000 7fbf3ed38700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2023-07-20T01:28:06.255361+0000 front 2023-07-20T01:28:06.255385+0000 (oldest deadline 2023-07-20T01:29:09.755472+0000) 2023-07-20T01:29:13.196 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:12 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:12.949+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:59.308083+0000 front 2023-07-20T01:27:59.308175+0000 (oldest deadline 2023-07-20T01:29:04.008156+0000) 2023-07-20T01:29:13.196 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:12 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:12.949+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2023-07-20T01:28:04.008814+0000 front 2023-07-20T01:28:04.008729+0000 (oldest deadline 2023-07-20T01:29:09.908820+0000) 2023-07-20T01:29:14.143 INFO:journalctl@ceph.osd.10.smithi136.stdout:Jul 20 01:29:13 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-10[119505]: 2023-07-20T01:29:13.685+0000 7fbf3ed38700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:57.653079+0000 front 2023-07-20T01:27:57.653056+0000 (oldest deadline 2023-07-20T01:29:01.153098+0000) 2023-07-20T01:29:14.143 INFO:journalctl@ceph.osd.10.smithi136.stdout:Jul 20 01:29:13 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-10[119505]: 2023-07-20T01:29:13.685+0000 7fbf3ed38700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2023-07-20T01:28:06.255361+0000 front 2023-07-20T01:28:06.255385+0000 (oldest deadline 2023-07-20T01:29:09.755472+0000) 2023-07-20T01:29:14.196 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:13 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:13.901+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:59.308083+0000 front 2023-07-20T01:27:59.308175+0000 (oldest deadline 2023-07-20T01:29:04.008156+0000) 2023-07-20T01:29:14.196 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:13 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:13.901+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2023-07-20T01:28:04.008814+0000 front 2023-07-20T01:28:04.008729+0000 (oldest deadline 2023-07-20T01:29:09.908820+0000) 2023-07-20T01:29:15.143 INFO:journalctl@ceph.osd.10.smithi136.stdout:Jul 20 01:29:14 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-10[119505]: 2023-07-20T01:29:14.735+0000 7fbf3ed38700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:57.653079+0000 front 2023-07-20T01:27:57.653056+0000 (oldest deadline 2023-07-20T01:29:01.153098+0000) 2023-07-20T01:29:15.143 INFO:journalctl@ceph.osd.10.smithi136.stdout:Jul 20 01:29:14 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-10[119505]: 2023-07-20T01:29:14.735+0000 7fbf3ed38700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2023-07-20T01:28:06.255361+0000 front 2023-07-20T01:28:06.255385+0000 (oldest deadline 2023-07-20T01:29:09.755472+0000) 2023-07-20T01:29:15.196 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:14 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:14.945+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:59.308083+0000 front 2023-07-20T01:27:59.308175+0000 (oldest deadline 2023-07-20T01:29:04.008156+0000) 2023-07-20T01:29:15.196 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:14 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:14.945+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2023-07-20T01:28:04.008814+0000 front 2023-07-20T01:28:04.008729+0000 (oldest deadline 2023-07-20T01:29:09.908820+0000) 2023-07-20T01:29:16.143 INFO:journalctl@ceph.osd.10.smithi136.stdout:Jul 20 01:29:15 smithi136 bash[125704]: ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-10 2023-07-20T01:29:16.446 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:15 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:15.994+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:59.308083+0000 front 2023-07-20T01:27:59.308175+0000 (oldest deadline 2023-07-20T01:29:04.008156+0000) 2023-07-20T01:29:16.446 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:15 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:15.994+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2023-07-20T01:28:04.008814+0000 front 2023-07-20T01:28:04.008729+0000 (oldest deadline 2023-07-20T01:29:09.908820+0000) 2023-07-20T01:29:17.196 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:16 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:16.947+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:59.308083+0000 front 2023-07-20T01:27:59.308175+0000 (oldest deadline 2023-07-20T01:29:04.008156+0000) 2023-07-20T01:29:17.196 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:16 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:16.947+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2023-07-20T01:28:04.008814+0000 front 2023-07-20T01:28:04.008729+0000 (oldest deadline 2023-07-20T01:29:09.908820+0000) 2023-07-20T01:29:17.405 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.10.service' 2023-07-20T01:29:17.485 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:29:17.486 INFO:tasks.cephadm.osd.10:Stopped osd.10 2023-07-20T01:29:17.487 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2023-07-20T01:29:17.487 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.11 2023-07-20T01:29:17.841 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:17 smithi183 systemd[1]: Stopping Ceph osd.11 for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:29:18.196 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:17 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:17.864+0000 7f49999c6700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:29:18.196 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:17 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:17.864+0000 7f49999c6700 -1 osd.11 82 *** Got signal Terminated *** 2023-07-20T01:29:18.197 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:17 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:17.864+0000 7f49999c6700 -1 osd.11 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-20T01:29:18.197 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:17 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:17.961+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:59.308083+0000 front 2023-07-20T01:27:59.308175+0000 (oldest deadline 2023-07-20T01:29:04.008156+0000) 2023-07-20T01:29:18.197 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:17 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:17.961+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2023-07-20T01:28:04.008814+0000 front 2023-07-20T01:28:04.008729+0000 (oldest deadline 2023-07-20T01:29:09.908820+0000) 2023-07-20T01:29:19.196 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:18 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:18.964+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:59.308083+0000 front 2023-07-20T01:27:59.308175+0000 (oldest deadline 2023-07-20T01:29:04.008156+0000) 2023-07-20T01:29:19.196 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:18 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:18.964+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2023-07-20T01:28:04.008814+0000 front 2023-07-20T01:28:04.008729+0000 (oldest deadline 2023-07-20T01:29:09.908820+0000) 2023-07-20T01:29:19.196 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:18 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:18.964+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2023-07-20T01:28:12.209794+0000 front 2023-07-20T01:28:12.209686+0000 (oldest deadline 2023-07-20T01:29:18.109777+0000) 2023-07-20T01:29:20.195 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:19 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:19.967+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:59.308083+0000 front 2023-07-20T01:27:59.308175+0000 (oldest deadline 2023-07-20T01:29:04.008156+0000) 2023-07-20T01:29:20.196 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:19 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:19.967+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2023-07-20T01:28:04.008814+0000 front 2023-07-20T01:28:04.008729+0000 (oldest deadline 2023-07-20T01:29:09.908820+0000) 2023-07-20T01:29:20.196 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:19 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:19.967+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2023-07-20T01:28:12.209794+0000 front 2023-07-20T01:28:12.209686+0000 (oldest deadline 2023-07-20T01:29:18.109777+0000) 2023-07-20T01:29:21.446 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:21 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:21.007+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:59.308083+0000 front 2023-07-20T01:27:59.308175+0000 (oldest deadline 2023-07-20T01:29:04.008156+0000) 2023-07-20T01:29:21.446 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:21 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:21.007+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2023-07-20T01:28:04.008814+0000 front 2023-07-20T01:28:04.008729+0000 (oldest deadline 2023-07-20T01:29:09.908820+0000) 2023-07-20T01:29:21.446 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:21 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:21.007+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2023-07-20T01:28:12.209794+0000 front 2023-07-20T01:28:12.209686+0000 (oldest deadline 2023-07-20T01:29:18.109777+0000) 2023-07-20T01:29:22.446 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:22 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:22.015+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2023-07-20T01:27:59.308083+0000 front 2023-07-20T01:27:59.308175+0000 (oldest deadline 2023-07-20T01:29:04.008156+0000) 2023-07-20T01:29:22.446 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:22 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:22.015+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.136:6804 osd.1 since back 2023-07-20T01:28:04.008814+0000 front 2023-07-20T01:28:04.008729+0000 (oldest deadline 2023-07-20T01:29:09.908820+0000) 2023-07-20T01:29:22.446 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:22 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11[120678]: 2023-07-20T01:29:22.015+0000 7f4995fbd700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.183:6804 osd.2 since back 2023-07-20T01:28:12.209794+0000 front 2023-07-20T01:28:12.209686+0000 (oldest deadline 2023-07-20T01:29:18.109777+0000) 2023-07-20T01:29:23.945 INFO:journalctl@ceph.osd.11.smithi183.stdout:Jul 20 01:29:23 smithi183 bash[127058]: ceph-7daf031e-269b-11ee-9b34-001a4aab830c-osd-11 2023-07-20T01:29:25.011 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@osd.11.service' 2023-07-20T01:29:25.051 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:29:25.051 INFO:tasks.cephadm.osd.11:Stopped osd.11 2023-07-20T01:29:25.052 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2023-07-20T01:29:25.052 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.a 2023-07-20T01:29:25.422 INFO:journalctl@ceph.mds.a.smithi060.stdout:Jul 20 01:29:25 smithi060 systemd[1]: Stopping Ceph mds.a for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:29:25.922 INFO:journalctl@ceph.mds.a.smithi060.stdout:Jul 20 01:29:25 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-a[131118]: 2023-07-20T01:29:25.451+0000 7f88510b0700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:29:25.923 INFO:journalctl@ceph.mds.a.smithi060.stdout:Jul 20 01:29:25 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-a[131118]: 2023-07-20T01:29:25.451+0000 7f88510b0700 -1 mds.a *** got signal Terminated *** 2023-07-20T01:29:27.079 INFO:journalctl@ceph.mds.a.smithi060.stdout:Jul 20 01:29:26 smithi060 bash[134557]: ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-a 2023-07-20T01:29:27.085 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.a.service' 2023-07-20T01:29:27.122 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:29:27.122 INFO:tasks.cephadm.mds.a:Stopped mds.a 2023-07-20T01:29:27.123 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2023-07-20T01:29:27.123 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.d 2023-07-20T01:29:27.422 INFO:journalctl@ceph.mds.d.smithi060.stdout:Jul 20 01:29:27 smithi060 systemd[1]: Stopping Ceph mds.d for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:29:27.922 INFO:journalctl@ceph.mds.d.smithi060.stdout:Jul 20 01:29:27 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-d[130121]: 2023-07-20T01:29:27.525+0000 7f955e8c4700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:29:27.923 INFO:journalctl@ceph.mds.d.smithi060.stdout:Jul 20 01:29:27 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-d[130121]: 2023-07-20T01:29:27.525+0000 7f955e8c4700 -1 mds.d *** got signal Terminated *** 2023-07-20T01:29:31.064 INFO:journalctl@ceph.mds.d.smithi060.stdout:Jul 20 01:29:30 smithi060 bash[134703]: ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-d 2023-07-20T01:29:31.071 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.d.service' 2023-07-20T01:29:31.108 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:29:31.108 INFO:tasks.cephadm.mds.d:Stopped mds.d 2023-07-20T01:29:31.109 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2023-07-20T01:29:31.109 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.g 2023-07-20T01:29:31.422 INFO:journalctl@ceph.mds.g.smithi060.stdout:Jul 20 01:29:31 smithi060 systemd[1]: Stopping Ceph mds.g for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:29:31.922 INFO:journalctl@ceph.mds.g.smithi060.stdout:Jul 20 01:29:31 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-g[130733]: 2023-07-20T01:29:31.516+0000 7fe1c358d700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:29:31.923 INFO:journalctl@ceph.mds.g.smithi060.stdout:Jul 20 01:29:31 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-g[130733]: 2023-07-20T01:29:31.516+0000 7fe1c358d700 -1 mds.g *** got signal Terminated *** 2023-07-20T01:29:35.922 INFO:journalctl@ceph.mds.g.smithi060.stdout:Jul 20 01:29:35 smithi060 bash[134841]: ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-g 2023-07-20T01:29:36.025 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.g.service' 2023-07-20T01:29:36.059 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:29:36.060 INFO:tasks.cephadm.mds.g:Stopped mds.g 2023-07-20T01:29:36.060 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2023-07-20T01:29:36.061 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.j 2023-07-20T01:29:36.422 INFO:journalctl@ceph.mds.j.smithi060.stdout:Jul 20 01:29:36 smithi060 systemd[1]: Stopping Ceph mds.j for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:29:36.923 INFO:journalctl@ceph.mds.j.smithi060.stdout:Jul 20 01:29:36 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-j[129735]: 2023-07-20T01:29:36.466+0000 7f125a930700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:29:36.923 INFO:journalctl@ceph.mds.j.smithi060.stdout:Jul 20 01:29:36 smithi060 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-j[129735]: 2023-07-20T01:29:36.466+0000 7f125a930700 -1 mds.j *** got signal Terminated *** 2023-07-20T01:29:40.922 INFO:journalctl@ceph.mds.j.smithi060.stdout:Jul 20 01:29:40 smithi060 bash[134985]: ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-j 2023-07-20T01:29:40.970 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.j.service' 2023-07-20T01:29:41.006 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:29:41.007 INFO:tasks.cephadm.mds.j:Stopped mds.j 2023-07-20T01:29:41.007 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2023-07-20T01:29:41.008 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.b 2023-07-20T01:29:41.366 INFO:journalctl@ceph.mds.b.smithi136.stdout:Jul 20 01:29:41 smithi136 systemd[1]: Stopping Ceph mds.b for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:29:41.643 INFO:journalctl@ceph.mds.b.smithi136.stdout:Jul 20 01:29:41 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-b[123171]: 2023-07-20T01:29:41.389+0000 7ff2be8ed700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:29:41.643 INFO:journalctl@ceph.mds.b.smithi136.stdout:Jul 20 01:29:41 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-b[123171]: 2023-07-20T01:29:41.389+0000 7ff2be8ed700 -1 mds.b *** got signal Terminated *** 2023-07-20T01:29:49.392 INFO:journalctl@ceph.mds.b.smithi136.stdout:Jul 20 01:29:49 smithi136 bash[125981]: ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-b 2023-07-20T01:29:49.437 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.b.service' 2023-07-20T01:29:49.473 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:29:49.474 INFO:tasks.cephadm.mds.b:Stopped mds.b 2023-07-20T01:29:49.474 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2023-07-20T01:29:49.475 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.e 2023-07-20T01:29:49.854 INFO:journalctl@ceph.mds.e.smithi136.stdout:Jul 20 01:29:49 smithi136 systemd[1]: Stopping Ceph mds.e for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:29:50.142 INFO:journalctl@ceph.mds.e.smithi136.stdout:Jul 20 01:29:49 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-e[122481]: 2023-07-20T01:29:49.878+0000 7f186903a700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:29:50.142 INFO:journalctl@ceph.mds.e.smithi136.stdout:Jul 20 01:29:49 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-e[122481]: 2023-07-20T01:29:49.878+0000 7f186903a700 -1 mds.e *** got signal Terminated *** 2023-07-20T01:29:56.544 INFO:journalctl@ceph.mds.e.smithi136.stdout:Jul 20 01:29:56 smithi136 bash[126121]: ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-e 2023-07-20T01:29:56.550 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.e.service' 2023-07-20T01:29:56.586 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:29:56.587 INFO:tasks.cephadm.mds.e:Stopped mds.e 2023-07-20T01:29:56.587 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2023-07-20T01:29:56.587 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.h 2023-07-20T01:29:56.892 INFO:journalctl@ceph.mds.h.smithi136.stdout:Jul 20 01:29:56 smithi136 systemd[1]: Stopping Ceph mds.h for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:29:57.392 INFO:journalctl@ceph.mds.h.smithi136.stdout:Jul 20 01:29:56 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-h[122821]: 2023-07-20T01:29:56.984+0000 7f3c73153700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:29:57.392 INFO:journalctl@ceph.mds.h.smithi136.stdout:Jul 20 01:29:56 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-h[122821]: 2023-07-20T01:29:56.984+0000 7f3c73153700 -1 mds.h *** got signal Terminated *** 2023-07-20T01:29:59.392 INFO:journalctl@ceph.mds.h.smithi136.stdout:Jul 20 01:29:59 smithi136 bash[126264]: ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-h 2023-07-20T01:29:59.558 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.h.service' 2023-07-20T01:29:59.592 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:29:59.593 INFO:tasks.cephadm.mds.h:Stopped mds.h 2023-07-20T01:29:59.593 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2023-07-20T01:29:59.593 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.k 2023-07-20T01:29:59.892 INFO:journalctl@ceph.mds.k.smithi136.stdout:Jul 20 01:29:59 smithi136 systemd[1]: Stopping Ceph mds.k for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:30:00.392 INFO:journalctl@ceph.mds.k.smithi136.stdout:Jul 20 01:30:00 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-k[122142]: 2023-07-20T01:30:00.002+0000 7f03cb8fd700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:30:00.393 INFO:journalctl@ceph.mds.k.smithi136.stdout:Jul 20 01:30:00 smithi136 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-k[122142]: 2023-07-20T01:30:00.002+0000 7f03cb8fd700 -1 mds.k *** got signal Terminated *** 2023-07-20T01:30:04.343 INFO:journalctl@ceph.mds.k.smithi136.stdout:Jul 20 01:30:04 smithi136 bash[126418]: ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-k 2023-07-20T01:30:04.542 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.k.service' 2023-07-20T01:30:04.585 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:30:04.586 INFO:tasks.cephadm.mds.k:Stopped mds.k 2023-07-20T01:30:04.587 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2023-07-20T01:30:04.587 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.c 2023-07-20T01:30:04.945 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 20 01:30:04 smithi183 systemd[1]: Stopping Ceph mds.c for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:30:05.445 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 20 01:30:04 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-c[124184]: 2023-07-20T01:30:04.980+0000 7f9210aed700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:30:05.445 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 20 01:30:04 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-c[124184]: 2023-07-20T01:30:04.980+0000 7f9210aed700 -1 mds.c *** got signal Terminated *** 2023-07-20T01:30:09.679 INFO:journalctl@ceph.mds.c.smithi183.stdout:Jul 20 01:30:09 smithi183 bash[127329]: ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-c 2023-07-20T01:30:09.686 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.c.service' 2023-07-20T01:30:09.724 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:30:09.725 INFO:tasks.cephadm.mds.c:Stopped mds.c 2023-07-20T01:30:09.725 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2023-07-20T01:30:09.727 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.f 2023-07-20T01:30:09.945 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 20 01:30:09 smithi183 systemd[1]: Stopping Ceph mds.f for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:30:10.445 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 20 01:30:10 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-f[123842]: 2023-07-20T01:30:10.129+0000 7f1969800700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:30:10.445 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 20 01:30:10 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-f[123842]: 2023-07-20T01:30:10.129+0000 7f1969800700 -1 mds.f *** got signal Terminated *** 2023-07-20T01:30:13.068 INFO:journalctl@ceph.mds.f.smithi183.stdout:Jul 20 01:30:13 smithi183 bash[127476]: ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-f 2023-07-20T01:30:13.256 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.f.service' 2023-07-20T01:30:13.290 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:30:13.290 INFO:tasks.cephadm.mds.f:Stopped mds.f 2023-07-20T01:30:13.291 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2023-07-20T01:30:13.291 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.i 2023-07-20T01:30:13.671 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 20 01:30:13 smithi183 systemd[1]: Stopping Ceph mds.i for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:30:13.945 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 20 01:30:13 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-i[124527]: 2023-07-20T01:30:13.696+0000 7f0b6589e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:30:13.945 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 20 01:30:13 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-i[124527]: 2023-07-20T01:30:13.696+0000 7f0b6589e700 -1 mds.i *** got signal Terminated *** 2023-07-20T01:30:18.422 INFO:journalctl@ceph.mds.i.smithi183.stdout:Jul 20 01:30:18 smithi183 bash[127614]: ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-i 2023-07-20T01:30:18.428 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.i.service' 2023-07-20T01:30:18.465 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:30:18.466 INFO:tasks.cephadm.mds.i:Stopped mds.i 2023-07-20T01:30:18.467 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2023-07-20T01:30:18.467 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.l 2023-07-20T01:30:18.695 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 20 01:30:18 smithi183 systemd[1]: Stopping Ceph mds.l for 7daf031e-269b-11ee-9b34-001a4aab830c... 2023-07-20T01:30:19.195 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 20 01:30:18 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-l[123496]: 2023-07-20T01:30:18.870+0000 7f421dd03700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-20T01:30:19.195 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 20 01:30:18 smithi183 ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-l[123496]: 2023-07-20T01:30:18.870+0000 7f421dd03700 -1 mds.l *** got signal Terminated *** 2023-07-20T01:30:20.695 INFO:journalctl@ceph.mds.l.smithi183.stdout:Jul 20 01:30:20 smithi183 bash[127769]: ceph-7daf031e-269b-11ee-9b34-001a4aab830c-mds-l 2023-07-20T01:30:20.815 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7daf031e-269b-11ee-9b34-001a4aab830c@mds.l.service' 2023-07-20T01:30:20.850 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-20T01:30:20.851 INFO:tasks.cephadm.mds.l:Stopped mds.l 2023-07-20T01:30:20.852 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7daf031e-269b-11ee-9b34-001a4aab830c --force --keep-logs 2023-07-20T01:30:21.175 INFO:teuthology.orchestra.run.smithi060.stdout:Deleting cluster with fsid: 7daf031e-269b-11ee-9b34-001a4aab830c 2023-07-20T01:30:22.818 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7daf031e-269b-11ee-9b34-001a4aab830c --force --keep-logs 2023-07-20T01:30:23.130 INFO:teuthology.orchestra.run.smithi136.stdout:Deleting cluster with fsid: 7daf031e-269b-11ee-9b34-001a4aab830c 2023-07-20T01:30:24.689 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7daf031e-269b-11ee-9b34-001a4aab830c --force --keep-logs 2023-07-20T01:30:24.988 INFO:teuthology.orchestra.run.smithi183.stdout:Deleting cluster with fsid: 7daf031e-269b-11ee-9b34-001a4aab830c 2023-07-20T01:30:26.543 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T01:30:26.580 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T01:30:26.617 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-20T01:30:26.647 INFO:tasks.cephadm:Archiving crash dumps... 2023-07-20T01:30:26.650 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344241/remote/smithi060/crash 2023-07-20T01:30:26.651 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/crash -- . 2023-07-20T01:30:26.695 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-20T01:30:26.696 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2023-07-20T01:30:26.697 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344241/remote/smithi136/crash 2023-07-20T01:30:26.699 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/crash -- . 2023-07-20T01:30:26.747 INFO:teuthology.orchestra.run.smithi136.stderr:tar: /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-20T01:30:26.748 INFO:teuthology.orchestra.run.smithi136.stderr:tar: Error is not recoverable: exiting now 2023-07-20T01:30:26.749 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344241/remote/smithi183/crash 2023-07-20T01:30:26.751 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/crash -- . 2023-07-20T01:30:26.782 INFO:teuthology.orchestra.run.smithi183.stderr:tar: /var/lib/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-20T01:30:26.783 INFO:teuthology.orchestra.run.smithi183.stderr:tar: Error is not recoverable: exiting now 2023-07-20T01:30:26.784 INFO:tasks.cephadm:Checking cluster log for badness... 2023-07-20T01:30:26.785 DEBUG:teuthology.orchestra.run.smithi060:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2023-07-20T01:30:26.820 INFO:teuthology.orchestra.run.smithi060.stderr:grep: /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph.log: No such file or directory 2023-07-20T01:30:26.822 INFO:tasks.cephadm:Compressing logs... 2023-07-20T01:30:26.822 DEBUG:teuthology.orchestra.run.smithi060:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-20T01:30:26.865 DEBUG:teuthology.orchestra.run.smithi136:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-20T01:30:26.867 DEBUG:teuthology.orchestra.run.smithi183:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-20T01:30:26.892 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-07-20T01:30:26.892 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-20T01:30:26.892 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.0.log 2023-07-20T01:30:26.893 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.g.log 2023-07-20T01:30:26.894 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.a.log 2023-07-20T01:30:26.894 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.0.log: /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.6.log 2023-07-20T01:30:26.894 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.3.log 2023-07-20T01:30:26.895 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.d.log 2023-07-20T01:30:26.895 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.3.log: /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.d.log: 94.9% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.g.log.gz 2023-07-20T01:30:26.895 INFO:teuthology.orchestra.run.smithi060.stderr: 95.0% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.a.log.gz 2023-07-20T01:30:26.896 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mgr.x.log 2023-07-20T01:30:26.896 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.9.log 2023-07-20T01:30:26.897 INFO:teuthology.orchestra.run.smithi183.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-20T01:30:26.897 INFO:teuthology.orchestra.run.smithi183.stderr:: No such file or directory 2023-07-20T01:30:26.897 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.5.log 2023-07-20T01:30:26.897 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.2.log 2023-07-20T01:30:26.897 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.11.log 2023-07-20T01:30:26.898 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.5.log: /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mgr.z.log 2023-07-20T01:30:26.898 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.8.log 92.0% 2023-07-20T01:30:26.898 INFO:teuthology.orchestra.run.smithi183.stderr: -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-20T01:30:26.898 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.f.log 2023-07-20T01:30:26.898 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.c.log 2023-07-20T01:30:26.899 INFO:teuthology.orchestra.run.smithi136.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-20T01:30:26.900 INFO:teuthology.orchestra.run.smithi136.stderr:: No such file or directory 2023-07-20T01:30:26.900 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.10.log 2023-07-20T01:30:26.900 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.h.log 2023-07-20T01:30:26.900 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.e.log 2023-07-20T01:30:26.900 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.4.log 2023-07-20T01:30:26.901 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.e.log: 91.6% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-20T01:30:26.901 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.1.log 2023-07-20T01:30:26.901 INFO:teuthology.orchestra.run.smithi136.stderr:gzip /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.4.log: -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.k.log 2023-07-20T01:30:26.901 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-volume.log 2023-07-20T01:30:26.902 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.f.log: /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.c.log: 91.3% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mgr.z.log.gz 2023-07-20T01:30:26.902 INFO:teuthology.orchestra.run.smithi183.stderr: 94.2% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.c.log.gz 2023-07-20T01:30:26.902 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.i.log 2023-07-20T01:30:26.902 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mgr.x.log: /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.9.log: 94.7% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.d.log.gz 2023-07-20T01:30:26.903 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-volume.log 2023-07-20T01:30:26.903 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-volume.log: 92.9% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-20T01:30:26.903 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.j.log 2023-07-20T01:30:26.903 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.j.log: 94.8% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.j.log.gz 2023-07-20T01:30:26.903 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mon.a.log 2023-07-20T01:30:26.904 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mon.c.log 2023-07-20T01:30:26.904 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.k.log: /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-volume.log: 93.3% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.h.log.gz 2023-07-20T01:30:26.905 INFO:teuthology.orchestra.run.smithi136.stderr: 93.9% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.k.log.gz 2023-07-20T01:30:26.905 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mon.a.log: 91.6% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-volume.log.gz 2023-07-20T01:30:26.907 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.i.log: /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mon.c.log: 93.6% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.f.log.gz 2023-07-20T01:30:26.907 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-volume.log 2023-07-20T01:30:26.909 INFO:teuthology.orchestra.run.smithi136.stderr: 91.5% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-volume.log.gz 2023-07-20T01:30:26.909 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.7.log 2023-07-20T01:30:26.909 INFO:teuthology.orchestra.run.smithi136.stderr: 93.0% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.e.log.gz 2023-07-20T01:30:26.910 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mon.b.log 2023-07-20T01:30:26.910 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.b.log 2023-07-20T01:30:26.911 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mgr.y.log 2023-07-20T01:30:26.911 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-volume.log: 93.9% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.i.log.gz 2023-07-20T01:30:26.912 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.l.log 2023-07-20T01:30:26.913 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.l.log: 95.2% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.l.log.gz 2023-07-20T01:30:26.913 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.b.log: /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mgr.y.log: 91.1% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mgr.y.log.gz 2023-07-20T01:30:26.914 INFO:teuthology.orchestra.run.smithi183.stderr: 91.5% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-volume.log.gz 2023-07-20T01:30:26.937 INFO:teuthology.orchestra.run.smithi136.stderr: 93.2% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mds.b.log.gz 2023-07-20T01:30:26.973 INFO:teuthology.orchestra.run.smithi183.stderr: 92.9% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.11.log.gz 2023-07-20T01:30:27.001 INFO:teuthology.orchestra.run.smithi060.stderr: 92.3% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.9.log.gz 2023-07-20T01:30:27.011 INFO:teuthology.orchestra.run.smithi136.stderr: 92.1% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mon.b.log.gz 2023-07-20T01:30:27.023 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mgr.x.log.gz 2023-07-20T01:30:27.026 INFO:teuthology.orchestra.run.smithi183.stderr: 92.1% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.8.log.gz 2023-07-20T01:30:27.029 INFO:teuthology.orchestra.run.smithi136.stderr: 92.1% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.4.log.gz 2023-07-20T01:30:27.030 INFO:teuthology.orchestra.run.smithi136.stderr: 92.2% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.10.log.gz 2023-07-20T01:30:27.031 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.3.log.gz 2023-07-20T01:30:27.035 INFO:teuthology.orchestra.run.smithi183.stderr: 92.0% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mon.c.log.gz 2023-07-20T01:30:27.042 INFO:teuthology.orchestra.run.smithi183.stderr: 91.9% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.5.log.gz 2023-07-20T01:30:27.052 INFO:teuthology.orchestra.run.smithi060.stderr: 92.1% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.6.log.gz 2023-07-20T01:30:27.087 INFO:teuthology.orchestra.run.smithi136.stderr: 92.0% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.7.log.gz 2023-07-20T01:30:27.098 INFO:teuthology.orchestra.run.smithi183.stderr: 91.9% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.2.log.gz 2023-07-20T01:30:27.100 INFO:teuthology.orchestra.run.smithi183.stderr: 2023-07-20T01:30:27.100 INFO:teuthology.orchestra.run.smithi183.stderr:real 0m0.216s 2023-07-20T01:30:27.100 INFO:teuthology.orchestra.run.smithi183.stderr:user 0m0.713s 2023-07-20T01:30:27.100 INFO:teuthology.orchestra.run.smithi183.stderr:sys 0m0.039s 2023-07-20T01:30:27.101 INFO:teuthology.orchestra.run.smithi060.stderr: 92.1% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.0.log.gz 2023-07-20T01:30:27.143 INFO:teuthology.orchestra.run.smithi136.stderr: 91.7% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-osd.1.log.gz 2023-07-20T01:30:27.145 INFO:teuthology.orchestra.run.smithi136.stderr: 2023-07-20T01:30:27.145 INFO:teuthology.orchestra.run.smithi136.stderr:real 0m0.261s 2023-07-20T01:30:27.146 INFO:teuthology.orchestra.run.smithi136.stderr:user 0m0.834s 2023-07-20T01:30:27.146 INFO:teuthology.orchestra.run.smithi136.stderr:sys 0m0.048s 2023-07-20T01:30:27.256 INFO:teuthology.orchestra.run.smithi060.stderr: 91.1% -- replaced with /var/log/ceph/7daf031e-269b-11ee-9b34-001a4aab830c/ceph-mon.a.log.gz 2023-07-20T01:30:27.258 INFO:teuthology.orchestra.run.smithi060.stderr: 2023-07-20T01:30:27.258 INFO:teuthology.orchestra.run.smithi060.stderr:real 0m0.378s 2023-07-20T01:30:27.258 INFO:teuthology.orchestra.run.smithi060.stderr:user 0m1.081s 2023-07-20T01:30:27.258 INFO:teuthology.orchestra.run.smithi060.stderr:sys 0m0.061s 2023-07-20T01:30:27.259 INFO:tasks.cephadm:Archiving logs... 2023-07-20T01:30:27.259 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344241/remote/smithi060/log 2023-07-20T01:30:27.260 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-20T01:30:27.454 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344241/remote/smithi136/log 2023-07-20T01:30:27.455 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-20T01:30:27.603 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344241/remote/smithi183/log 2023-07-20T01:30:27.606 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-20T01:30:27.729 INFO:tasks.cephadm:Removing cluster... 2023-07-20T01:30:27.730 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7daf031e-269b-11ee-9b34-001a4aab830c --force 2023-07-20T01:30:28.025 INFO:teuthology.orchestra.run.smithi060.stdout:Deleting cluster with fsid: 7daf031e-269b-11ee-9b34-001a4aab830c 2023-07-20T01:30:28.494 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7daf031e-269b-11ee-9b34-001a4aab830c --force 2023-07-20T01:30:28.796 INFO:teuthology.orchestra.run.smithi136.stdout:Deleting cluster with fsid: 7daf031e-269b-11ee-9b34-001a4aab830c 2023-07-20T01:30:29.247 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7daf031e-269b-11ee-9b34-001a4aab830c --force 2023-07-20T01:30:29.548 INFO:teuthology.orchestra.run.smithi183.stdout:Deleting cluster with fsid: 7daf031e-269b-11ee-9b34-001a4aab830c 2023-07-20T01:30:29.998 INFO:tasks.cephadm:Removing cephadm ... 2023-07-20T01:30:29.998 DEBUG:teuthology.orchestra.run.smithi060:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-20T01:30:30.020 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-20T01:30:30.040 DEBUG:teuthology.orchestra.run.smithi183:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-20T01:30:30.059 INFO:tasks.cephadm:Teardown complete 2023-07-20T01:30:30.060 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephadm.py", line 1643, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephadm.py", line 1558, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephadm.py", line 980, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/filesystem.py", line 535, in __init__ self.create() File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/filesystem.py", line 773, in create subvol_path = self.get_ceph_cmd_stdout( File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/cephfs_test_case.py", line 73, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/cephfs_test_case.py", line 58, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 2: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name=qa sv_0' 2023-07-20T01:30:30.136 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=37032b232f624782b9f7de4ef8c50eb7 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephadm.py", line 1643, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephadm.py", line 1558, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephadm.py", line 980, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/filesystem.py", line 535, in __init__ self.create() File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/filesystem.py", line 773, in create subvol_path = self.get_ceph_cmd_stdout( File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/cephfs_test_case.py", line 73, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/cephfs/cephfs_test_case.py", line 58, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_1b0b2eb005a7c51d97a788b9900d511e1394b537/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 2: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name=qa sv_0' 2023-07-20T01:30:30.141 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-07-20T01:30:30.150 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-07-20T01:30:30.164 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-07-20T01:30:30.165 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-20T01:30:30.191 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-20T01:30:30.195 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-20T01:30:30.308 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-07-20T01:30:30.316 INFO:teuthology.task.clock:Checking final clock skew... 2023-07-20T01:30:30.317 DEBUG:teuthology.orchestra.run.smithi060:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T01:30:30.319 DEBUG:teuthology.orchestra.run.smithi136:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T01:30:30.322 DEBUG:teuthology.orchestra.run.smithi183:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-20T01:30:30.336 INFO:teuthology.orchestra.run.smithi060.stderr:bash: ntpq: command not found 2023-07-20T01:30:30.337 INFO:teuthology.orchestra.run.smithi183.stderr:bash: ntpq: command not found 2023-07-20T01:30:30.338 INFO:teuthology.orchestra.run.smithi136.stderr:bash: ntpq: command not found 2023-07-20T01:30:30.341 INFO:teuthology.orchestra.run.smithi060.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T01:30:30.341 INFO:teuthology.orchestra.run.smithi060.stdout:=============================================================================== 2023-07-20T01:30:30.341 INFO:teuthology.orchestra.run.smithi060.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 48 -2116us[-2079us] +/- 86ms 2023-07-20T01:30:30.341 INFO:teuthology.orchestra.run.smithi060.stdout:^* hv02.front.sepia.ceph.com 3 6 377 48 -2568us[-2530us] +/- 64ms 2023-07-20T01:30:30.342 INFO:teuthology.orchestra.run.smithi060.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 47 +4778us[+4778us] +/- 76ms 2023-07-20T01:30:30.342 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T01:30:30.343 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T01:30:30.344 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2023-07-20T01:30:30.344 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 49 -2089us[-2089us] +/- 86ms 2023-07-20T01:30:30.344 INFO:teuthology.orchestra.run.smithi136.stdout:^* hv02.front.sepia.ceph.com 3 6 377 50 -2539us[-2535us] +/- 64ms 2023-07-20T01:30:30.344 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 49 +4789us[+4789us] +/- 76ms 2023-07-20T01:30:30.345 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T01:30:30.345 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-20T01:30:30.345 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2023-07-20T01:30:30.345 INFO:teuthology.orchestra.run.smithi183.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 48 -2070us[-2070us] +/- 86ms 2023-07-20T01:30:30.345 INFO:teuthology.orchestra.run.smithi183.stdout:^* hv02.front.sepia.ceph.com 3 6 377 49 -2516us[-2521us] +/- 64ms 2023-07-20T01:30:30.346 INFO:teuthology.orchestra.run.smithi183.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 47 +4821us[+4821us] +/- 76ms 2023-07-20T01:30:30.346 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-20T01:30:30.347 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-07-20T01:30:30.356 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-07-20T01:30:30.357 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-07-20T01:30:30.392 DEBUG:teuthology.orchestra.run.smithi060:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-20T01:30:30.494 DEBUG:teuthology.orchestra.run.smithi136:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-20T01:30:30.595 DEBUG:teuthology.orchestra.run.smithi183:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-20T01:30:30.686 DEBUG:teuthology.orchestra.run.smithi060:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T01:30:30.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T01:30:30.721 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T01:30:30.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T01:30:30.755 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-20T01:30:30.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T01:30:30.787 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-07-20T01:30:30.796 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-07-20T01:30:30.806 INFO:teuthology.task.internal:Duration was 1740.313566 seconds 2023-07-20T01:30:30.806 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-07-20T01:30:30.816 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-07-20T01:30:30.817 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-20T01:30:30.820 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-20T01:30:30.822 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-20T01:30:30.867 INFO:teuthology.orchestra.run.smithi060.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T01:30:30.868 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T01:30:30.872 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-20T01:30:31.185 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-07-20T01:30:31.186 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi060.front.sepia.ceph.com 2023-07-20T01:30:31.186 DEBUG:teuthology.orchestra.run.smithi060:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-20T01:30:31.210 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi136.front.sepia.ceph.com 2023-07-20T01:30:31.211 DEBUG:teuthology.orchestra.run.smithi136:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-20T01:30:31.232 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi183.front.sepia.ceph.com 2023-07-20T01:30:31.233 DEBUG:teuthology.orchestra.run.smithi183:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-20T01:30:31.255 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-07-20T01:30:31.256 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-20T01:30:31.258 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-20T01:30:31.275 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-20T01:30:31.341 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-07-20T01:30:31.342 DEBUG:teuthology.orchestra.run.smithi060:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-20T01:30:31.344 DEBUG:teuthology.orchestra.run.smithi136:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-20T01:30:31.360 DEBUG:teuthology.orchestra.run.smithi183:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-20T01:30:31.464 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-07-20T01:30:31.481 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-07-20T01:30:31.482 DEBUG:teuthology.orchestra.run.smithi060:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-20T01:30:31.510 DEBUG:teuthology.orchestra.run.smithi136:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-20T01:30:31.537 DEBUG:teuthology.orchestra.run.smithi183:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-20T01:30:31.564 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-07-20T01:30:31.580 DEBUG:teuthology.orchestra.run.smithi060:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-20T01:30:31.582 DEBUG:teuthology.orchestra.run.smithi136:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-20T01:30:31.585 DEBUG:teuthology.orchestra.run.smithi183:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-20T01:30:31.607 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = core 2023-07-20T01:30:31.608 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = core 2023-07-20T01:30:31.630 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = core 2023-07-20T01:30:31.648 DEBUG:teuthology.orchestra.run.smithi060:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-20T01:30:31.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T01:30:31.695 DEBUG:teuthology.orchestra.run.smithi136:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-20T01:30:31.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T01:30:31.712 DEBUG:teuthology.orchestra.run.smithi183:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-20T01:30:31.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-20T01:30:31.728 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-07-20T01:30:31.738 INFO:teuthology.task.internal:Transferring archived files... 2023-07-20T01:30:31.739 DEBUG:teuthology.misc:Transferring archived files from smithi060:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344241/remote/smithi060 2023-07-20T01:30:31.740 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-20T01:30:31.790 DEBUG:teuthology.misc:Transferring archived files from smithi136:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344241/remote/smithi136 2023-07-20T01:30:31.791 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-20T01:30:31.828 DEBUG:teuthology.misc:Transferring archived files from smithi183:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344241/remote/smithi183 2023-07-20T01:30:31.829 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-20T01:30:31.871 INFO:teuthology.task.internal:Removing archive directory... 2023-07-20T01:30:31.871 DEBUG:teuthology.orchestra.run.smithi060:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-20T01:30:31.874 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-20T01:30:31.876 DEBUG:teuthology.orchestra.run.smithi183:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-20T01:30:31.921 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-07-20T01:30:31.932 INFO:teuthology.task.internal:Not uploading archives. 2023-07-20T01:30:31.933 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-07-20T01:30:31.941 INFO:teuthology.task.internal:Tidying up after the test... 2023-07-20T01:30:31.942 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-20T01:30:31.944 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-20T01:30:31.946 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-20T01:30:31.958 INFO:teuthology.orchestra.run.smithi060.stdout: 262307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 20 01:30 /home/ubuntu/cephtest 2023-07-20T01:30:31.960 INFO:teuthology.orchestra.run.smithi136.stdout: 262307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 20 01:30 /home/ubuntu/cephtest 2023-07-20T01:30:31.977 INFO:teuthology.orchestra.run.smithi183.stdout: 262170 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 20 01:30 /home/ubuntu/cephtest 2023-07-20T01:30:31.978 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-07-20T01:30:31.989 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-07-20T01:30:32.009 INFO:teuthology.nuke:Checking targets against current locks 2023-07-20T01:30:32.038 DEBUG:teuthology.nuke:shortname: smithi060 2023-07-20T01:30:32.039 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-20T01:30:32.065 DEBUG:teuthology.nuke:shortname: smithi136 2023-07-20T01:30:32.065 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-20T01:30:32.070 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi060.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344241', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 00:56:00.541788', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-20T01:30:32.087 DEBUG:teuthology.nuke:shortname: smithi183 2023-07-20T01:30:32.088 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-20T01:30:32.095 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344241', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 00:56:00.545106', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-20T01:30:32.109 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-20_00:39:06-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7344241', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 00:56:00.543826', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-20T01:30:32.125 INFO:teuthology.orchestra.console:Power off smithi060 2023-07-20T01:30:32.126 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-20T01:30:32.149 INFO:teuthology.orchestra.console:Power off smithi136 2023-07-20T01:30:32.150 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-20T01:30:32.164 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T01:30:32.180 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T01:30:32.200 INFO:teuthology.orchestra.console:Power off smithi183 2023-07-20T01:30:32.201 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-20T01:30:32.225 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T01:30:36.191 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T01:30:36.214 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T01:30:36.250 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T01:30:40.308 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T01:30:40.327 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T01:30:40.362 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T01:30:40.520 INFO:teuthology.orchestra.console:Power off for smithi060 completed 2023-07-20T01:30:40.577 INFO:teuthology.orchestra.console:Power off for smithi183 completed 2023-07-20T01:30:44.444 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-20T01:30:44.657 INFO:teuthology.orchestra.console:Power off for smithi136 completed 2023-07-20T01:30:44.764 INFO:teuthology.run:Summary data: description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} duration: 1740.3135659694672 failure_reason: 'Command failed on smithi060 with status 2: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name=qa sv_0''' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=37032b232f624782b9f7de4ef8c50eb7 status: fail success: false 2023-07-20T01:30:44.765 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-20T01:30:44.866 INFO:teuthology.run:FAIL